Jul 28 15:32:42.382825 ip-10-0-143-228 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 15:32:42.781865 ip-10-0-143-228 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:32:42.781865 ip-10-0-143-228 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 15:32:42.781865 ip-10-0-143-228 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:32:42.781865 ip-10-0-143-228 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 15:32:42.781865 ip-10-0-143-228 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:32:42.782743 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.782660    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 15:32:42.790852 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790830    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:42.790852 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790848    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:42.790852 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790852    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:42.790852 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790855    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:42.790852 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790858    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:42.790852 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790861    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790865    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790868    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790871    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790874    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790877    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790880    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790883    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790885    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790888    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790890    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790893    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790903    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790906    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790908    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790911    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790914    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790916    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790919    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790922    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:42.791078 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790924    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790927    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790930    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790933    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790936    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790939    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790941    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790944    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790947    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790950    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790952    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790955    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790958    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790960    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790964    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790967    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790970    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790973    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790976    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790979    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:42.791547 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790981    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790984    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790986    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790989    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790991    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790994    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790996    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.790999    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791001    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791004    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791006    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791009    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791013    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791016    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791018    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791021    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791024    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791026    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791029    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791032    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:42.792062 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791034    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791038    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791043    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791046    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791049    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791051    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791054    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791057    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791059    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791062    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791064    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791067    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791070    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791073    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791075    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791078    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791082    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791087    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791097    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:42.792534 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791100    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791103    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791106    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791109    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791113    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791115    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791118    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791121    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791124    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791127    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791129    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791133    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791136    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791139    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791142    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791871    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791878    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791881    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791884    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791888    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:42.793004 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791890    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791893    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791896    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791899    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791902    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791905    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791907    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791910    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791913    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791916    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791919    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791923    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791926    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791929    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791932    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791934    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791937    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791939    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791942    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791945    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:42.793479 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791947    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791950    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791953    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791955    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791958    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791961    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791963    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791966    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791969    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791971    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791974    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791977    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791981    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791984    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791987    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791990    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791994    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.791997    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792000    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:42.793998 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792003    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792006    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792009    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792011    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792014    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792017    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792020    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792022    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792025    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792028    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792031    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792034    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792037    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792040    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792043    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792048    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792051    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792054    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792057    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792060    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:42.794461 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792063    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792066    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792069    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792072    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792075    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792079    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792082    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792084    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792087    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792089    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792092    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792095    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792097    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792100    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792103    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792105    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792108    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792110    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792113    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:42.794980 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792116    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792119    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792121    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792124    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792126    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792129    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792132    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792134    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792137    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792140    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792142    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792146    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792149    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792152    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792154    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792157    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792544    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792548    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792551    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:42.795436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792570    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792573    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792576    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792580    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792583    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792586    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792588    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792591    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792594    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792596    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792599    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792602    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792604    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792608    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792610    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792613    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792616    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792619    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792621    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792624    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:42.795907 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792627    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792629    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792632    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792636    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792640    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792644    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792647    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792650    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792652    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792655    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792658    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792661    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792664    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792666    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792669    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792672    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792675    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792678    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792680    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:42.796412 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792683    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792685    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792688    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792691    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792694    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792696    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792699    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792705    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792708    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792711    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792713    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792716    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792718    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792721    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792723    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792726    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792728    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792731    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792734    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792736    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:42.796877 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792739    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792741    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792744    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792746    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792749    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792751    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792754    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792757    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792759    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792762    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792764    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792767    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792769    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792771    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792774    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792778    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792781    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792784    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792786    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:42.797382 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792789    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792797    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792802    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792805    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792808    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792811    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792814    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792817    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792820    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792822    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792825    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792828    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792831    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792833    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792836    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792838    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792841    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.792844    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792911    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792918    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792930    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 15:32:42.797858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792937    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792943    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792948    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792955    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792959    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792962    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792965    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792969    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792972    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792975    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792978    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792981    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792984    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792989    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792992    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792996    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.792999    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793002    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793005    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793009    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793013    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793016    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793019    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 15:32:42.798354 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793022    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793026    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793029    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793031    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793034    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793037    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793041    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793044    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793047    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793050    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793053    2575 flags.go:64] FLAG: --enable-server="true"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793056    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793060    2575 flags.go:64] FLAG: --event-burst="100"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793063    2575 flags.go:64] FLAG: --event-qps="50"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793066    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793069    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793071    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793075    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793078    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793080    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793083    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793086    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793090    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793094    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793097    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 15:32:42.798970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793100    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793102    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793105    2575 flags.go:64] FLAG: --feature-gates=""
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793110    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793113    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793116    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793119    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793123    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793125    2575 flags.go:64] FLAG: --help="false"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793128    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-143-228.ec2.internal"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793131    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793134    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793137    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793141    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793144    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793147    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793150    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793153    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793155    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793158    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793161    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793164    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793167    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793170    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 15:32:42.799601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793173    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793176    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793179    2575 flags.go:64] FLAG: --lock-file=""
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793181    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793184    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793188    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793194    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793198    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793201    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793203    2575 flags.go:64] FLAG: --logging-format="text"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793206    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793211    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793213    2575 flags.go:64] FLAG: --manifest-url=""
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793216    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793221    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793224    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793228    2575 flags.go:64] FLAG: --max-pods="110"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793231    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793234    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793237    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793240    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793243    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793246    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793249    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793256    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 15:32:42.800174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793259    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793262    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793265    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793268    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793273    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793276    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793279    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793282    2575 flags.go:64] FLAG: --port="10250"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793285    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793288    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e18a1db8dbe4744a"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793291    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793294    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793298    2575 flags.go:64] FLAG: --register-node="true"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793301    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793305    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793308    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793311    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793314    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793317    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793321    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793324    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793327    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793330    2575 flags.go:64] FLAG: --runonce="false"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793333    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793336    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 15:32:42.800866 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793339    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793342    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793345    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793348    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793351    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793354    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793357    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793360    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793363    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793365    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793368    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793371    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793374    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793379    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793382    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793385    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793389    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793392    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793395    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793399    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793402    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793405    2575 flags.go:64] FLAG: --v="2"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793409    2575 flags.go:64] FLAG: --version="false"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793413    2575 flags.go:64] FLAG: --vmodule=""
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793417    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 15:32:42.801472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793420    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793541    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793545    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793548    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793551    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793565    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793568    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793571    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793574    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793577    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793579    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793583    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793585    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793588    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793591    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793594    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793596    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793599    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793601    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:42.802104 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793606    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793609    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793612    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793615    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793619    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793621    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793624    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793629    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793632    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793635    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793637    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793640    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793643    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793646    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793651    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793653    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793656    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793659    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793662    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:42.802635 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793664    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793667    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793670    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793672    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793675    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793678    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793680    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793683    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793685    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793688    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793691    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793693    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793696    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793698    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793701    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793704    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793706    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793710    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793714    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:42.803113 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793718    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793724    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793727    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793730    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793733    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793735    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793738    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793741    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793746    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793749    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793752    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793754    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793757    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793760    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793762    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793765    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793768    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793771    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793774    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793776    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:42.803590 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793779    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793781    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793784    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793786    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793789    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793792    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793795    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793797    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793800    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793802    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793805    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793807    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793810    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793814    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793817    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793819    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793822    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793824    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793827    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793830    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:42.804081 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793834    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793837    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.793839    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.793845    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.800387    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.800403    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.800425    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.800431    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800486    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800491    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800494    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800497    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800500    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800503    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800506    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:42.804567 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800508    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800511    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800514    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800517    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800519    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800522    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800524    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800528    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800530    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800533    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800536    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800538    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800541    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800545    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800549    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800552    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800568    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800571    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800574    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800577    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:42.804962 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800579    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800582    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800587    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800592    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800595    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800598    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800601    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800604    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800607    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800609    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800612    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800615    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800618    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800621    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800623    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800626    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800628    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800631    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800633    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800636    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:42.805436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800639    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800642    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800644    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800647    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800649    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800652    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800654    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800657    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800659    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800662    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800665    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800667    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800670    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800672    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800676    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800679    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800683    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800685    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800688    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800691    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:42.805953 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800694    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800696    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800699    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800701    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800704    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800707    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800710    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800712    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800715    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800717    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800720    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800724    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800728    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800731    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800734    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800737    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800740    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800743    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800745    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:42.806442 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800748    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800750    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800753    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800755    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800758    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800761    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800763    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800766    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800769    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800772    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800775    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800778    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800780    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.800785    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800895    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:32:42.807059 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800900    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800903    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800906    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800908    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800911    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800914    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800916    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800919    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800921    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800924    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800928    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800932    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800935    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800938    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800941    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800943    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800946    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800949    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800951    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800954    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:32:42.807443 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800956    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800959    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800961    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800964    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800967    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800970    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800972    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800975    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800978    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800980    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800983    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800985    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800988    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800990    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800993    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800995    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.800998    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801000    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801003    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:32:42.807935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801006    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801008    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801012    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801014    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801017    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801020    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801023    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801025    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801027    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801030    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801032    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801036    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801039    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801041    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801044    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801046    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801049    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801052    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801054    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:32:42.808391 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801057    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801059    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801062    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801065    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801067    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801069    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801072    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801074    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801077    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801080    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801082    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801084    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801087    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801090    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801092    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801095    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801098    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801100    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801103    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801105    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:32:42.808856 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801108    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801111    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801115    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801118    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801121    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801123    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801126    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801129    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801131    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801134    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801137    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801140    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801143    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801146    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801148    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801151    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801154    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801157    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801159    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:32:42.809380 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:42.801162    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:32:42.810045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.801167    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:32:42.810045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.801258    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 15:32:42.810045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.803395    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 15:32:42.810045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.804815    2575 server.go:1019] "Starting client certificate rotation"
Jul 28 15:32:42.810045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.804916    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 15:32:42.810045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.804949    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 15:32:42.810045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.807644    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 15:32:42.810045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.808853    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 15:32:42.819299 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.819280    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 28 15:32:42.821377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.821362    2575 log.go:25] "Validated CRI v1 image API"
Jul 28 15:32:42.825192 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.825176    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 15:32:42.827594 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.827570    2575 fs.go:135] Filesystem UUIDs: map[6490f044-2405-4044-aaa0-4ab1f69152a1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 edb20ed4-2de0-4420-bebe-3d82c752dad7:/dev/nvme0n1p3]
Jul 28 15:32:42.827594 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.827589    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 15:32:42.833822 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.833712    2575 manager.go:216] Machine: {Timestamp:2026-07-28 15:32:42.832370274 +0000 UTC m=+0.334034065 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100118 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21365bbf26a7d0724953ee6e683c0c SystemUUID:ec21365b-bf26-a7d0-7249-53ee6e683c0c BootID:d3ad3c18-2bf9-4255-b39f-7105beb20a62 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:de:0b:05:02:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:de:0b:05:02:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:a1:d6:06:79:1b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 15:32:42.833822 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.833815    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 15:32:42.833953 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.833918    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 15:32:42.837013 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.836984    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 15:32:42.837077 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.837062    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 15:32:42.837169 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.837015    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 15:32:42.837701 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.837691    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 15:32:42.837738 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.837703    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 15:32:42.837738 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.837710    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 15:32:42.838304 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.838294    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 15:32:42.839577 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.839567    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 15:32:42.842855 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.842844    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 15:32:42.843006 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.842996    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 15:32:42.843040 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.843025    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 15:32:42.843040 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.843034    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 15:32:42.843089 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.843048    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 15:32:42.843089 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.843056    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 28 15:32:42.843089 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.843065    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 15:32:42.843380 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.843368    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 15:32:42.843421 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.843388    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 15:32:42.847224 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.847205    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 15:32:42.848583 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.848552    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 15:32:42.848691 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.848606    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 15:32:42.849311 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849297    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 15:32:42.849377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849319    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 15:32:42.849377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849328    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 15:32:42.849377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849336    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 15:32:42.849377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849349    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 15:32:42.849377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849358    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 15:32:42.849377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849370    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 15:32:42.849377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849378    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 15:32:42.849620 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849389    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 15:32:42.849620 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849398    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 15:32:42.849620 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849410    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 15:32:42.849620 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.849419    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 15:32:42.850170 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.850158    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 15:32:42.850222 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.850172    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 15:32:42.853087 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.853071    2575 server.go:1268] "Started kubelet"
Jul 28 15:32:42.853169 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.853130    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 15:32:42.853325 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.853274    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 15:32:42.853372 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.853348    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 15:32:42.853760 ip-10-0-143-228 systemd[1]: Started Kubernetes Kubelet.
Jul 28 15:32:42.857162 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.857140    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 15:32:42.857260 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.857228    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 15:32:42.857407 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.857384    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 15:32:42.857407 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.857383    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 15:32:42.859320 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.859295    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 15:32:42.860888 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.860869    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 15:32:42.863605 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.862469    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-228.ec2.internal.18c67deddc23f32a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-228.ec2.internal,UID:ip-10-0-143-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-228.ec2.internal,},FirstTimestamp:2026-07-28 15:32:42.853045034 +0000 UTC m=+0.354708826,LastTimestamp:2026-07-28 15:32:42.853045034 +0000 UTC m=+0.354708826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-228.ec2.internal,}"
Jul 28 15:32:42.866427 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.866408    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 15:32:42.866427 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.866423    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 15:32:42.866573 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.866534    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:42.868200 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.867486    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 15:32:42.868414 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868398    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 15:32:42.868499 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868418    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 15:32:42.868598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868498    2575 factory.go:55] Registering systemd factory
Jul 28 15:32:42.868598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868512    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 28 15:32:42.868598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868536    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 15:32:42.868598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868544    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 15:32:42.868784 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868703    2575 factory.go:153] Registering CRI-O factory
Jul 28 15:32:42.868784 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868712    2575 factory.go:223] Registration of the crio container factory successfully
Jul 28 15:32:42.868784 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868758    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 15:32:42.868784 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868784    2575 factory.go:103] Registering Raw factory
Jul 28 15:32:42.868974 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.868799    2575 manager.go:1196] Started watching for new ooms in manager
Jul 28 15:32:42.869875 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.869853    2575 manager.go:349] Starting recovery of all containers
Jul 28 15:32:42.877313 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.877292    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 15:32:42.877401 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.877377    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 15:32:42.880252 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.880240    2575 manager.go:354] Recovery completed
Jul 28 15:32:42.884335 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.884323    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:42.886097 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.886079    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:42.886194 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.886111    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:42.886194 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.886125    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:42.886609 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.886591    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 15:32:42.886609 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.886609    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:42.886735 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.886610    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 15:32:42.886735 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.886698    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 15:32:42.887259 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.887245    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:42.887326 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.887272    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:42.887326 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.887286    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:42.887951 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.887844    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-228.ec2.internal.18c67dedde1c4524  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-228.ec2.internal,UID:ip-10-0-143-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-228.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-228.ec2.internal,},FirstTimestamp:2026-07-28 15:32:42.886096164 +0000 UTC m=+0.387759962,LastTimestamp:2026-07-28 15:32:42.886096164 +0000 UTC m=+0.387759962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-228.ec2.internal,}"
Jul 28 15:32:42.888055 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.887968    2575 policy_none.go:49] "None policy: Start"
Jul 28 15:32:42.888055 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.887984    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 15:32:42.888055 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.887997    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 15:32:42.889331 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.889320    2575 policy_none.go:47] "Start"
Jul 28 15:32:42.891484 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.891467    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lbbwl"
Jul 28 15:32:42.897445 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.897382    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-228.ec2.internal.18c67dedde1c9940  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-228.ec2.internal,UID:ip-10-0-143-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-228.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-228.ec2.internal,},FirstTimestamp:2026-07-28 15:32:42.886117696 +0000 UTC m=+0.387781491,LastTimestamp:2026-07-28 15:32:42.886117696 +0000 UTC m=+0.387781491,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-228.ec2.internal,}"
Jul 28 15:32:42.898252 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.898234    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lbbwl"
Jul 28 15:32:42.921205 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.921066    2575 manager.go:337] "Starting Device Plugin manager"
Jul 28 15:32:42.921284 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.921216    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 15:32:42.921284 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.921229    2575 server.go:87] "Starting device plugin registration server"
Jul 28 15:32:42.921440 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.921428    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 15:32:42.921490 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.921442    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 15:32:42.921536 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.921520    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 15:32:42.921627 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.921610    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 15:32:42.921627 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.921622    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 15:32:42.922735 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.922716    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 15:32:42.922807 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.922758    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:42.997814 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.997783    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 15:32:42.998949 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.998933    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 15:32:42.999026 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.998976    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 15:32:42.999026 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:42.999003    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 15:32:42.999117 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:42.999042    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 15:32:43.000940 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:43.000921    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:43.021675 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:43.021659    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:43.022415 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:43.022401    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:43.022476 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:43.022428    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:43.022476 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:43.022438    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:43.022476 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:43.022457    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-228.ec2.internal"
Jul 28 15:32:43.031789 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:43.031774    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-228.ec2.internal"
Jul 28 15:32:43.031861 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:43.031793    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-228.ec2.internal\": node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:43.057966 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:43.057915    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:44.351265 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.351240    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal"]
Jul 28 15:32:44.351334 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.351305    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:44.352355 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.352341    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:44.352426 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.352368    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:44.352426 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.352377    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:44.353607 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.353595    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:44.353720 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.353707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.353765 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.353739    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:44.354450 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.354434    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:44.354542 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.354458    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:44.354542 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.354468    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:44.354542 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.354491    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:44.354542 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.354521    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:44.354542 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.354535    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:44.355720 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.355707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.355781 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.355732    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:32:44.356490 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.356476    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:32:44.356558 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.356497    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:32:44.356558 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.356512    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:32:44.384824 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:44.384805    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-228.ec2.internal\" not found" node="ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.389072 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:44.389057    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-228.ec2.internal\" not found" node="ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.410477 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:44.410459    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:44.421506 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.421487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d9f3368b81aaba73b18a11f9ce1769e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"3d9f3368b81aaba73b18a11f9ce1769e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.421596 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.421512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6807966616aea208bee6841100daf208-config\") pod \"kube-apiserver-proxy-ip-10-0-143-228.ec2.internal\" (UID: \"6807966616aea208bee6841100daf208\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.421596 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.421530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d9f3368b81aaba73b18a11f9ce1769e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"3d9f3368b81aaba73b18a11f9ce1769e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.510817 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:44.510792    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:44.522240 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.522221    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6807966616aea208bee6841100daf208-config\") pod \"kube-apiserver-proxy-ip-10-0-143-228.ec2.internal\" (UID: \"6807966616aea208bee6841100daf208\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.522326 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.522244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d9f3368b81aaba73b18a11f9ce1769e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"3d9f3368b81aaba73b18a11f9ce1769e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.522326 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.522262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d9f3368b81aaba73b18a11f9ce1769e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"3d9f3368b81aaba73b18a11f9ce1769e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.522326 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.522306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d9f3368b81aaba73b18a11f9ce1769e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"3d9f3368b81aaba73b18a11f9ce1769e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.522449 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.522331    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d9f3368b81aaba73b18a11f9ce1769e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"3d9f3368b81aaba73b18a11f9ce1769e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.522449 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.522352    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6807966616aea208bee6841100daf208-config\") pod \"kube-apiserver-proxy-ip-10-0-143-228.ec2.internal\" (UID: \"6807966616aea208bee6841100daf208\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.611633 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:44.611557    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:44.687108 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.687077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.692344 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.692328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:44.712294 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:44.712273    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:44.812855 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:44.812824    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:44.913394 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:44.913337    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found"
Jul 28 15:32:44.943457 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:44.943437    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:45.019292 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.019268    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:45.029681 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.029659    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 15:32:45.031460 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.031446    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"
Jul 28 15:32:45.043956 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.043936    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 15:32:45.056085 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.056067    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 15:32:45.056228 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.056212    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 15:32:45.056278 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.056225    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 15:32:45.079910 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.079887    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:45.095435 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.095410    2575 apiserver.go:52] "Watching apiserver"
Jul 28 15:32:45.104810 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.104787    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 15:32:45.106312 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.106289    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-5zfcm","openshift-network-diagnostics/network-check-target-q7wx2","openshift-network-operator/iptables-alerter-zvqkp","kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal","openshift-cluster-node-tuning-operator/tuned-kbmm9","openshift-dns/node-resolver-rr2gh","openshift-multus/multus-9cqsh","openshift-multus/network-metrics-daemon-lx8xf","openshift-ovn-kubernetes/ovnkube-node-fk7vh","kube-system/global-pull-secret-syncer-f5ms2","kube-system/konnectivity-agent-s7dpj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2","openshift-image-registry/node-ca-8v5vg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal"]
Jul 28 15:32:45.107800 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.107780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.107892 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.107870    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:32:45.108812 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.108791    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.110206 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.110188    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.111345 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.111325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:45.111435 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.111391    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 15:32:45.111490 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.111392    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:32:45.111490 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.111396    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 15:32:45.111688 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.111670    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-67rnw\""
Jul 28 15:32:45.111790 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.111713    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 15:32:45.112612 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.112591    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.112807 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.112592    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 15:32:45.112962 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.112946    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 15:32:45.112962 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.112959    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:32:45.113079 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.113015    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2dz8v\""
Jul 28 15:32:45.113612 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.113597    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.114867 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.114852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.115432 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.115417    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 15:32:45.115524 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.115435    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jr77w\""
Jul 28 15:32:45.115524 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.115449    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:32:45.116278 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.116260    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.116412 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.116395    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 15:32:45.116412 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.116408    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 15:32:45.116754 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.116742    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2ttjn\""
Jul 28 15:32:45.117565 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.117552    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:45.117615 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.117596    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:32:45.118010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.117992    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 15:32:45.118072 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.118010    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 15:32:45.118072 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.117992    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 15:32:45.118072 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.118027    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 15:32:45.118072 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.118029    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 15:32:45.118236 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.118126    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 15:32:45.119017 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.118995    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 15:32:45.119017 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.119011    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p2xp7\""
Jul 28 15:32:45.119433 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.119413    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.119521 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.119470    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7zj8r\""
Jul 28 15:32:45.119957 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.119936    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 15:32:45.120564 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.120370    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 15:32:45.120564 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.120475    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 15:32:45.120678 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.120619    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 15:32:45.121856 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.121835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:32:45.123510 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.122285    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 15:32:45.123510 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.122429    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vcb52\""
Jul 28 15:32:45.123510 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.122677    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.123510 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.123009    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 15:32:45.125196 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125129    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 15:32:45.125286 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125247    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7rbpc\""
Jul 28 15:32:45.125598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125505    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 15:32:45.125760 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysctl-d\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.125830 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125764    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 15:32:45.125830 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125770    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 15:32:45.125949 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125770    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:45.125949 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125866    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-system-cni-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.125949 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-cni-binary-copy\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.126107 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.125959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-netns\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.126107 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126021    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-cni-multus\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.126107 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-hostroot\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.126290 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-system-cni-dir\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.126290 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hns\" (UniqueName: \"kubernetes.io/projected/9ab5b87b-81cc-4429-8147-c990db710694-kube-api-access-l9hns\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:45.126290 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysconfig\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.126290 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-slash\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.126290 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.126537 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126297    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:45.126537 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126459    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2dj48\""
Jul 28 15:32:45.126637 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.126536    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 15:32:45.127120 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.127206 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-cnibin\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.127255 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-run\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.127255 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-sys\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.127255 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127249    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-conf-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.127373 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4wq\" (UniqueName: \"kubernetes.io/projected/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-kube-api-access-ql4wq\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.127373 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127299    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysctl-conf\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.127373 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-lib-modules\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.127373 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127360    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8qd\" (UniqueName: \"kubernetes.io/projected/8014099c-4821-4606-bf04-5517c49877e7-kube-api-access-2q8qd\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.127513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-etc-kubernetes\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.127513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127410    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.127513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127435    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjk7b\" (UniqueName: \"kubernetes.io/projected/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-kube-api-access-zjk7b\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.127513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127461    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.127513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-var-lib-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127511    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-env-overrides\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127717 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcnv\" (UniqueName: \"kubernetes.io/projected/74285376-f5c6-4e10-a103-9fd61945f750-kube-api-access-6vcnv\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127717 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrk7\" (UniqueName: \"kubernetes.io/projected/5c0a5568-31d1-4001-b029-47265e985078-kube-api-access-5mrk7\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.127717 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127586    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-host\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.127717 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-hosts-file\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.127717 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-host-slash\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.127717 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-kubelet\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127717 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-systemd-units\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127742    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74285376-f5c6-4e10-a103-9fd61945f750-ovn-node-metrics-cert\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c0a5568-31d1-4001-b029-47265e985078-serviceca\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127772    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-modprobe-d\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-systemd\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-k8s-cni-cncf-io\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127830    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-cni-bin\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127844    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-daemon-config\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127858    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-kubelet\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-iptables-alerter-script\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-systemd\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-ovn\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-ovnkube-config\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.127978 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.127981    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128031    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-run-netns\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-ovnkube-script-lib\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128115    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c75abded-e781-47c2-a233-e50561257d20-kubelet-config\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd64t\" (UniqueName: \"kubernetes.io/projected/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-kube-api-access-xd64t\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128191    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-kubernetes\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-var-lib-kubelet\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128237    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cnibin\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-os-release\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-tmp-dir\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-os-release\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-socket-dir-parent\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-multus-certs\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128444    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-node-log\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.128534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c75abded-e781-47c2-a233-e50561257d20-dbus\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-cni-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128511    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlf7\" (UniqueName: \"kubernetes.io/projected/38e5b545-9292-4266-bdc0-63ba5e9c70e3-kube-api-access-zqlf7\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128534    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-cni-bin\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c0a5568-31d1-4001-b029-47265e985078-host\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-etc-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8014099c-4821-4606-bf04-5517c49877e7-etc-tuned\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8014099c-4821-4606-bf04-5517c49877e7-tmp\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128705    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-log-socket\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.129048 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.128723    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-cni-netd\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.132998 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.132981    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 15:32:45.153059 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.153002    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:27:44 +0000 UTC" deadline="2028-02-08 02:49:42.843827444 +0000 UTC"
Jul 28 15:32:45.153059 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.153057    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13427h16m57.690773s"
Jul 28 15:32:45.155837 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.155728    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sn84g"
Jul 28 15:32:45.163690 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.163646    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sn84g"
Jul 28 15:32:45.208221 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.208198    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6807966616aea208bee6841100daf208.slice/crio-39e19bbc880c0ee7bbfdf8e412fcf6d5c052111c069de13d593f10a76f4d1bd3 WatchSource:0}: Error finding container 39e19bbc880c0ee7bbfdf8e412fcf6d5c052111c069de13d593f10a76f4d1bd3: Status 404 returned error can't find the container with id 39e19bbc880c0ee7bbfdf8e412fcf6d5c052111c069de13d593f10a76f4d1bd3
Jul 28 15:32:45.212946 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.212932    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:32:45.219942 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.219924    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 15:32:45.228857 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.228840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysctl-d\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.228949 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.228867    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:45.228949 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.228891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-system-cni-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.228949 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.228915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-cni-binary-copy\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229099 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.228984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-system-cni-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229099 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.228994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-netns\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229099 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.229010    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:45.229099 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229023    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysctl-d\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.229099 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-cni-multus\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229099 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-netns\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229099 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.229095    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs podName:9ab5b87b-81cc-4429-8147-c990db710694 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:45.729053359 +0000 UTC m=+1.979110358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs") pod "network-metrics-daemon-lx8xf" (UID: "9ab5b87b-81cc-4429-8147-c990db710694") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-cni-multus\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-hostroot\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-system-cni-dir\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229211    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-hostroot\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229246    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hns\" (UniqueName: \"kubernetes.io/projected/9ab5b87b-81cc-4429-8147-c990db710694-kube-api-access-l9hns\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229288    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysconfig\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-slash\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229357    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-system-cni-dir\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:45.229388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysconfig\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229395    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229405    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-slash\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-device-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229475    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.229485    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229517    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-cni-binary-copy\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229529    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-cnibin\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-cnibin\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.229566    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret podName:c75abded-e781-47c2-a233-e50561257d20 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:45.729550776 +0000 UTC m=+1.979607763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret") pod "global-pull-secret-syncer-f5ms2" (UID: "c75abded-e781-47c2-a233-e50561257d20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-run\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-sys\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-conf-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4wq\" (UniqueName: \"kubernetes.io/projected/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-kube-api-access-ql4wq\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysctl-conf\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-conf-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229718    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-lib-modules\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-run\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.229874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229742    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8qd\" (UniqueName: \"kubernetes.io/projected/8014099c-4821-4606-bf04-5517c49877e7-kube-api-access-2q8qd\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-etc-kubernetes\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-sys\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjk7b\" (UniqueName: \"kubernetes.io/projected/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-kube-api-access-zjk7b\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229847    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-lib-modules\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-var-lib-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229892    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-etc-kubernetes\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-env-overrides\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcnv\" (UniqueName: \"kubernetes.io/projected/74285376-f5c6-4e10-a103-9fd61945f750-kube-api-access-6vcnv\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229962    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-var-lib-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.229996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrk7\" (UniqueName: \"kubernetes.io/projected/5c0a5568-31d1-4001-b029-47265e985078-kube-api-access-5mrk7\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-host\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-hosts-file\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-host-slash\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.230676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-kubelet\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230302    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-host-slash\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230318    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-hosts-file\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-systemd-units\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230356    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-host\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-kubelet\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230372    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-systemd-units\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-env-overrides\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230426    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74285376-f5c6-4e10-a103-9fd61945f750-ovn-node-metrics-cert\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c0a5568-31d1-4001-b029-47265e985078-serviceca\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230474    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230474    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-modprobe-d\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230527    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-systemd\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-sysctl-conf\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.231571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230568    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-modprobe-d\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-k8s-cni-cncf-io\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230597    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-systemd\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-cni-bin\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-k8s-cni-cncf-io\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230631    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-cni-bin\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230641    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-daemon-config\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0d40f3e-c36b-4def-acaa-dcf458c09666-agent-certs\") pod \"konnectivity-agent-s7dpj\" (UID: \"d0d40f3e-c36b-4def-acaa-dcf458c09666\") " pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-socket-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-kubelet\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-iptables-alerter-script\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230755    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-systemd\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-var-lib-kubelet\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-systemd\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-ovn\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230914    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c0a5568-31d1-4001-b029-47265e985078-serviceca\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-ovnkube-config\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.232374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.230994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-run-netns\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-ovnkube-script-lib\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c75abded-e781-47c2-a233-e50561257d20-kubelet-config\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231120    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-registration-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231142    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd64t\" (UniqueName: \"kubernetes.io/projected/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-kube-api-access-xd64t\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-kubernetes\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-var-lib-kubelet\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cnibin\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-os-release\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-tmp-dir\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-os-release\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-socket-dir-parent\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-multus-certs\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.233257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231362    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-iptables-alerter-script\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-node-log\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-run-ovn\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231431    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-ovnkube-config\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231155    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-daemon-config\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231477    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c75abded-e781-47c2-a233-e50561257d20-dbus\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c75abded-e781-47c2-a233-e50561257d20-dbus\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231500    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c75abded-e781-47c2-a233-e50561257d20-kubelet-config\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231529    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cnibin\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-socket-dir-parent\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231565    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-os-release\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231580    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-host-run-multus-certs\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231604    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-var-lib-kubelet\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231609    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-node-log\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-cni-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8014099c-4821-4606-bf04-5517c49877e7-etc-kubernetes\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlf7\" (UniqueName: \"kubernetes.io/projected/38e5b545-9292-4266-bdc0-63ba5e9c70e3-kube-api-access-zqlf7\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.233725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-cni-bin\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74285376-f5c6-4e10-a103-9fd61945f750-ovnkube-script-lib\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0d40f3e-c36b-4def-acaa-dcf458c09666-konnectivity-ca\") pod \"konnectivity-agent-s7dpj\" (UID: \"d0d40f3e-c36b-4def-acaa-dcf458c09666\") " pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231755    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-run-netns\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-sys-fs\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231808    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-multus-cni-dir\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c0a5568-31d1-4001-b029-47265e985078-host\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231872    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-os-release\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.231951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38e5b545-9292-4266-bdc0-63ba5e9c70e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232016    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-cni-bin\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-etc-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzpz\" (UniqueName: \"kubernetes.io/projected/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-kube-api-access-5vzpz\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232121    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-tmp-dir\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232134    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c0a5568-31d1-4001-b029-47265e985078-host\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232221    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8014099c-4821-4606-bf04-5517c49877e7-etc-tuned\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8014099c-4821-4606-bf04-5517c49877e7-tmp\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-log-socket\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.234531 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-cni-netd\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.235305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-host-cni-netd\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.235305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-log-socket\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.235305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232222    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74285376-f5c6-4e10-a103-9fd61945f750-etc-openvswitch\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.235305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.232675    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38e5b545-9292-4266-bdc0-63ba5e9c70e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.235305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.234641    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8014099c-4821-4606-bf04-5517c49877e7-tmp\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.235305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.234848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8014099c-4821-4606-bf04-5517c49877e7-etc-tuned\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.236706 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.235898    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74285376-f5c6-4e10-a103-9fd61945f750-ovn-node-metrics-cert\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.245994 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.245979    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:45.246057 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.245998    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:45.246057 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.246008    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9l5c9 for pod openshift-network-diagnostics/network-check-target-q7wx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:45.246121 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.246060    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9 podName:a6fe2c5a-6be6-4a10-82af-f895ddcdfda2 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:45.74604843 +0000 UTC m=+1.996105413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5c9" (UniqueName: "kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9") pod "network-check-target-q7wx2" (UID: "a6fe2c5a-6be6-4a10-82af-f895ddcdfda2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:45.249358 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.249335    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4wq\" (UniqueName: \"kubernetes.io/projected/f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1-kube-api-access-ql4wq\") pod \"multus-9cqsh\" (UID: \"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1\") " pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.249460 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.249446    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hns\" (UniqueName: \"kubernetes.io/projected/9ab5b87b-81cc-4429-8147-c990db710694-kube-api-access-l9hns\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:45.254196 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.254144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" event={"ID":"6807966616aea208bee6841100daf208","Type":"ContainerStarted","Data":"39e19bbc880c0ee7bbfdf8e412fcf6d5c052111c069de13d593f10a76f4d1bd3"}
Jul 28 15:32:45.259480 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.259455    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjk7b\" (UniqueName: \"kubernetes.io/projected/bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4-kube-api-access-zjk7b\") pod \"iptables-alerter-zvqkp\" (UID: \"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4\") " pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.259755 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.259741    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcnv\" (UniqueName: \"kubernetes.io/projected/74285376-f5c6-4e10-a103-9fd61945f750-kube-api-access-6vcnv\") pod \"ovnkube-node-fk7vh\" (UID: \"74285376-f5c6-4e10-a103-9fd61945f750\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.264005 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.263981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrk7\" (UniqueName: \"kubernetes.io/projected/5c0a5568-31d1-4001-b029-47265e985078-kube-api-access-5mrk7\") pod \"node-ca-8v5vg\" (UID: \"5c0a5568-31d1-4001-b029-47265e985078\") " pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.264124 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.264104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlf7\" (UniqueName: \"kubernetes.io/projected/38e5b545-9292-4266-bdc0-63ba5e9c70e3-kube-api-access-zqlf7\") pod \"multus-additional-cni-plugins-5zfcm\" (UID: \"38e5b545-9292-4266-bdc0-63ba5e9c70e3\") " pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.264210 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.264127    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd64t\" (UniqueName: \"kubernetes.io/projected/30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882-kube-api-access-xd64t\") pod \"node-resolver-rr2gh\" (UID: \"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882\") " pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.264261 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.264204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8qd\" (UniqueName: \"kubernetes.io/projected/8014099c-4821-4606-bf04-5517c49877e7-kube-api-access-2q8qd\") pod \"tuned-kbmm9\" (UID: \"8014099c-4821-4606-bf04-5517c49877e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.333024 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-sys-fs\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333126 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzpz\" (UniqueName: \"kubernetes.io/projected/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-kube-api-access-5vzpz\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333126 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-device-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333256 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333124    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-sys-fs\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333256 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-device-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333256 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333256 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0d40f3e-c36b-4def-acaa-dcf458c09666-agent-certs\") pod \"konnectivity-agent-s7dpj\" (UID: \"d0d40f3e-c36b-4def-acaa-dcf458c09666\") " pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:32:45.333256 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-socket-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333478 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333478 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-registration-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333478 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333478 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333350    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333478 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333364    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-registration-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333478 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0d40f3e-c36b-4def-acaa-dcf458c09666-konnectivity-ca\") pod \"konnectivity-agent-s7dpj\" (UID: \"d0d40f3e-c36b-4def-acaa-dcf458c09666\") " pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:32:45.333478 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333418    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-socket-dir\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.333895 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.333879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0d40f3e-c36b-4def-acaa-dcf458c09666-konnectivity-ca\") pod \"konnectivity-agent-s7dpj\" (UID: \"d0d40f3e-c36b-4def-acaa-dcf458c09666\") " pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:32:45.335174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.335144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0d40f3e-c36b-4def-acaa-dcf458c09666-agent-certs\") pod \"konnectivity-agent-s7dpj\" (UID: \"d0d40f3e-c36b-4def-acaa-dcf458c09666\") " pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:32:45.340729 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.340708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzpz\" (UniqueName: \"kubernetes.io/projected/45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f-kube-api-access-5vzpz\") pod \"aws-ebs-csi-driver-node-jq2x2\" (UID: \"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.391339 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.391320    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9f3368b81aaba73b18a11f9ce1769e.slice/crio-9ffdf73ec7702f4ebe6f807a2eefcdf30ce3d27ddca31019ac10ba52027f0e6b WatchSource:0}: Error finding container 9ffdf73ec7702f4ebe6f807a2eefcdf30ce3d27ddca31019ac10ba52027f0e6b: Status 404 returned error can't find the container with id 9ffdf73ec7702f4ebe6f807a2eefcdf30ce3d27ddca31019ac10ba52027f0e6b
Jul 28 15:32:45.429935 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.429890    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8v5vg"
Jul 28 15:32:45.437408 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.437390    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0a5568_31d1_4001_b029_47265e985078.slice/crio-46917b8fe87e4adf9a1b134dad7f72ec1f90289df126a0f9a80407d514b49bf3 WatchSource:0}: Error finding container 46917b8fe87e4adf9a1b134dad7f72ec1f90289df126a0f9a80407d514b49bf3: Status 404 returned error can't find the container with id 46917b8fe87e4adf9a1b134dad7f72ec1f90289df126a0f9a80407d514b49bf3
Jul 28 15:32:45.454379 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.454359    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zvqkp"
Jul 28 15:32:45.461293 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.461269    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf75d18f_4f83_4f4d_8dc5_5f8e3e7987d4.slice/crio-d48ac2357b241222395c855abda7b7790df3e354218e89430daad726d3ff5cd9 WatchSource:0}: Error finding container d48ac2357b241222395c855abda7b7790df3e354218e89430daad726d3ff5cd9: Status 404 returned error can't find the container with id d48ac2357b241222395c855abda7b7790df3e354218e89430daad726d3ff5cd9
Jul 28 15:32:45.467174 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.467143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kbmm9"
Jul 28 15:32:45.474422 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.474405    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8014099c_4821_4606_bf04_5517c49877e7.slice/crio-7cd0525103edc46501b86f51d592b6b46079ac6b9e85080cd4c44dc7bfe1653d WatchSource:0}: Error finding container 7cd0525103edc46501b86f51d592b6b46079ac6b9e85080cd4c44dc7bfe1653d: Status 404 returned error can't find the container with id 7cd0525103edc46501b86f51d592b6b46079ac6b9e85080cd4c44dc7bfe1653d
Jul 28 15:32:45.492001 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.491978    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rr2gh"
Jul 28 15:32:45.498512 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.498486    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a8b0f0_8c63_4b32_a1e5_8c2b4bcd9882.slice/crio-a766e8051631280239e74a7e4d7f042669c2d45222db72c4c70a7cb6a09afbd5 WatchSource:0}: Error finding container a766e8051631280239e74a7e4d7f042669c2d45222db72c4c70a7cb6a09afbd5: Status 404 returned error can't find the container with id a766e8051631280239e74a7e4d7f042669c2d45222db72c4c70a7cb6a09afbd5
Jul 28 15:32:45.512586 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.512567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:32:45.517709 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.517693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9cqsh"
Jul 28 15:32:45.519463 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.519447    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74285376_f5c6_4e10_a103_9fd61945f750.slice/crio-722ea354b072f3a4c9edf7b366147cbbf20fbac1c223e8d7f963ed0f50d08a2e WatchSource:0}: Error finding container 722ea354b072f3a4c9edf7b366147cbbf20fbac1c223e8d7f963ed0f50d08a2e: Status 404 returned error can't find the container with id 722ea354b072f3a4c9edf7b366147cbbf20fbac1c223e8d7f963ed0f50d08a2e
Jul 28 15:32:45.523234 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.523220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5zfcm"
Jul 28 15:32:45.524542 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.524524    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf571f6f2_c1d0_4ef1_a0b4_3df8e717b9b1.slice/crio-60d11cb5a74e5c9787d666ff8719169f7b7c9c0ffd263d30fe8c64556fde6306 WatchSource:0}: Error finding container 60d11cb5a74e5c9787d666ff8719169f7b7c9c0ffd263d30fe8c64556fde6306: Status 404 returned error can't find the container with id 60d11cb5a74e5c9787d666ff8719169f7b7c9c0ffd263d30fe8c64556fde6306
Jul 28 15:32:45.528739 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.528723    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:32:45.531499 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.531476    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e5b545_9292_4266_bdc0_63ba5e9c70e3.slice/crio-e60350236bab9e3d66bfc5011394db86b40bfb593d1346dbaa3ad1da93aac3d7 WatchSource:0}: Error finding container e60350236bab9e3d66bfc5011394db86b40bfb593d1346dbaa3ad1da93aac3d7: Status 404 returned error can't find the container with id e60350236bab9e3d66bfc5011394db86b40bfb593d1346dbaa3ad1da93aac3d7
Jul 28 15:32:45.534989 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.534973    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2"
Jul 28 15:32:45.535222 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.535202    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d40f3e_c36b_4def_acaa_dcf458c09666.slice/crio-c71dd45c948f3a3a15db1d6c5199e23346381638a1fdb97f594d06ba895b9fe1 WatchSource:0}: Error finding container c71dd45c948f3a3a15db1d6c5199e23346381638a1fdb97f594d06ba895b9fe1: Status 404 returned error can't find the container with id c71dd45c948f3a3a15db1d6c5199e23346381638a1fdb97f594d06ba895b9fe1
Jul 28 15:32:45.542840 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:32:45.542823    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e0a2a1_1e0c_4bdd_9aad_97e624f8de6f.slice/crio-54a4b20031616d12d826ee3e0b402cf7903741e5ee5ca02c0a413bba163da0c8 WatchSource:0}: Error finding container 54a4b20031616d12d826ee3e0b402cf7903741e5ee5ca02c0a413bba163da0c8: Status 404 returned error can't find the container with id 54a4b20031616d12d826ee3e0b402cf7903741e5ee5ca02c0a413bba163da0c8
Jul 28 15:32:45.559782 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.559764    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:45.736850 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.736765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:45.736850 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.736833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:45.737361 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.737319    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:45.737481 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.737411    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret podName:c75abded-e781-47c2-a233-e50561257d20 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:46.73737988 +0000 UTC m=+2.987436885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret") pod "global-pull-secret-syncer-f5ms2" (UID: "c75abded-e781-47c2-a233-e50561257d20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:45.739006 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.738974    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:45.739112 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.739051    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs podName:9ab5b87b-81cc-4429-8147-c990db710694 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:46.739034204 +0000 UTC m=+2.989091201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs") pod "network-metrics-daemon-lx8xf" (UID: "9ab5b87b-81cc-4429-8147-c990db710694") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:45.837904 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:45.837649    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:45.837904 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.837821    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:45.837904 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.837840    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:45.837904 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.837852    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9l5c9 for pod openshift-network-diagnostics/network-check-target-q7wx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:45.837904 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:45.837907    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9 podName:a6fe2c5a-6be6-4a10-82af-f895ddcdfda2 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:46.83788847 +0000 UTC m=+3.087945454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5c9" (UniqueName: "kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9") pod "network-check-target-q7wx2" (UID: "a6fe2c5a-6be6-4a10-82af-f895ddcdfda2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:46.044548 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.044466    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:46.166211 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.166130    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:27:45 +0000 UTC" deadline="2028-04-13 19:35:18.59492578 +0000 UTC"
Jul 28 15:32:46.166211 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.166207    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15004h2m32.428722657s"
Jul 28 15:32:46.253630 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.253602    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:46.253804 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.253721    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:32:46.285210 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.285144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerStarted","Data":"e60350236bab9e3d66bfc5011394db86b40bfb593d1346dbaa3ad1da93aac3d7"}
Jul 28 15:32:46.301032 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.300963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" event={"ID":"3d9f3368b81aaba73b18a11f9ce1769e","Type":"ContainerStarted","Data":"9ffdf73ec7702f4ebe6f807a2eefcdf30ce3d27ddca31019ac10ba52027f0e6b"}
Jul 28 15:32:46.309423 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.309393    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7dpj" event={"ID":"d0d40f3e-c36b-4def-acaa-dcf458c09666","Type":"ContainerStarted","Data":"c71dd45c948f3a3a15db1d6c5199e23346381638a1fdb97f594d06ba895b9fe1"}
Jul 28 15:32:46.317090 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.317064    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9cqsh" event={"ID":"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1","Type":"ContainerStarted","Data":"60d11cb5a74e5c9787d666ff8719169f7b7c9c0ffd263d30fe8c64556fde6306"}
Jul 28 15:32:46.336505 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.336474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"722ea354b072f3a4c9edf7b366147cbbf20fbac1c223e8d7f963ed0f50d08a2e"}
Jul 28 15:32:46.358385 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.358356    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rr2gh" event={"ID":"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882","Type":"ContainerStarted","Data":"a766e8051631280239e74a7e4d7f042669c2d45222db72c4c70a7cb6a09afbd5"}
Jul 28 15:32:46.361414 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.361388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kbmm9" event={"ID":"8014099c-4821-4606-bf04-5517c49877e7","Type":"ContainerStarted","Data":"7cd0525103edc46501b86f51d592b6b46079ac6b9e85080cd4c44dc7bfe1653d"}
Jul 28 15:32:46.375827 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.374602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zvqkp" event={"ID":"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4","Type":"ContainerStarted","Data":"d48ac2357b241222395c855abda7b7790df3e354218e89430daad726d3ff5cd9"}
Jul 28 15:32:46.381767 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.381715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8v5vg" event={"ID":"5c0a5568-31d1-4001-b029-47265e985078","Type":"ContainerStarted","Data":"46917b8fe87e4adf9a1b134dad7f72ec1f90289df126a0f9a80407d514b49bf3"}
Jul 28 15:32:46.385358 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.385335    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2" event={"ID":"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f","Type":"ContainerStarted","Data":"54a4b20031616d12d826ee3e0b402cf7903741e5ee5ca02c0a413bba163da0c8"}
Jul 28 15:32:46.750273 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.750191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:46.750273 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.750257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:46.750483 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.750411    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:46.750483 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.750469    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret podName:c75abded-e781-47c2-a233-e50561257d20 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:48.750452058 +0000 UTC m=+5.000509045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret") pod "global-pull-secret-syncer-f5ms2" (UID: "c75abded-e781-47c2-a233-e50561257d20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:46.750866 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.750844    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:46.750946 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.750902    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs podName:9ab5b87b-81cc-4429-8147-c990db710694 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:48.750885559 +0000 UTC m=+5.000942549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs") pod "network-metrics-daemon-lx8xf" (UID: "9ab5b87b-81cc-4429-8147-c990db710694") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:46.851497 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:46.850796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:46.851497 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.850993    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:46.851497 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.851018    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:46.851497 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.851032    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9l5c9 for pod openshift-network-diagnostics/network-check-target-q7wx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:46.851497 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:46.851089    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9 podName:a6fe2c5a-6be6-4a10-82af-f895ddcdfda2 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:48.851071762 +0000 UTC m=+5.101128767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5c9" (UniqueName: "kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9") pod "network-check-target-q7wx2" (UID: "a6fe2c5a-6be6-4a10-82af-f895ddcdfda2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:47.166780 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:47.166691    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:27:45 +0000 UTC" deadline="2028-02-28 08:16:42.524523442 +0000 UTC"
Jul 28 15:32:47.166780 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:47.166734    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13912h43m55.357793904s"
Jul 28 15:32:47.251143 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:47.251112    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:47.251329 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:47.251251    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:32:47.251663 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:47.251642    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:47.251775 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:47.251754    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:32:47.427976 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:47.427892    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:32:48.253576 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:48.253551    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:48.253985 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.253672    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:32:48.773267 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:48.773227    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:48.773463 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:48.773302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:48.773463 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.773434    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:48.773580 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.773502    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret podName:c75abded-e781-47c2-a233-e50561257d20 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:52.773481174 +0000 UTC m=+9.023538185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret") pod "global-pull-secret-syncer-f5ms2" (UID: "c75abded-e781-47c2-a233-e50561257d20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:48.773902 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.773882    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:48.773989 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.773940    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs podName:9ab5b87b-81cc-4429-8147-c990db710694 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:52.773923707 +0000 UTC m=+9.023980697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs") pod "network-metrics-daemon-lx8xf" (UID: "9ab5b87b-81cc-4429-8147-c990db710694") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:48.874132 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:48.874030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:48.874322 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.874219    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:48.874322 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.874243    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:48.874322 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.874258    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9l5c9 for pod openshift-network-diagnostics/network-check-target-q7wx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:48.874480 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:48.874327    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9 podName:a6fe2c5a-6be6-4a10-82af-f895ddcdfda2 nodeName:}" failed. No retries permitted until 2026-07-28 15:32:52.874308768 +0000 UTC m=+9.124365769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5c9" (UniqueName: "kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9") pod "network-check-target-q7wx2" (UID: "a6fe2c5a-6be6-4a10-82af-f895ddcdfda2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:49.251985 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:49.251360    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:49.251985 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:49.251393    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:49.251985 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:49.251514    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:32:49.251985 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:49.251897    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:32:50.251081 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:50.251052    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:50.252088 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:50.251191    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:32:51.251571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:51.251511    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:51.251571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:51.251537    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:51.252069 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:51.251645    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:32:51.252069 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:51.251963    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:32:52.251179 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:52.251075    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:52.251341 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.251252    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:32:52.807600 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:52.807251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:52.807600 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:52.807320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:52.807600 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.807498    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:52.807600 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.807514    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:52.807600 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.807567    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret podName:c75abded-e781-47c2-a233-e50561257d20 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:00.807546153 +0000 UTC m=+17.057603157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret") pod "global-pull-secret-syncer-f5ms2" (UID: "c75abded-e781-47c2-a233-e50561257d20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:32:52.807600 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.807586    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs podName:9ab5b87b-81cc-4429-8147-c990db710694 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:00.807576244 +0000 UTC m=+17.057633232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs") pod "network-metrics-daemon-lx8xf" (UID: "9ab5b87b-81cc-4429-8147-c990db710694") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:32:52.908077 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:52.907982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:52.908258 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.908147    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:32:52.908258 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.908183    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:32:52.908258 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.908196    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9l5c9 for pod openshift-network-diagnostics/network-check-target-q7wx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:52.908258 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:52.908249    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9 podName:a6fe2c5a-6be6-4a10-82af-f895ddcdfda2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:00.908232898 +0000 UTC m=+17.158289898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5c9" (UniqueName: "kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9") pod "network-check-target-q7wx2" (UID: "a6fe2c5a-6be6-4a10-82af-f895ddcdfda2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:32:53.251389 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:53.251353    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:53.251617 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:53.251498    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:32:53.251617 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:53.251549    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:53.251743 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:53.251688    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:32:54.252548 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:54.252517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:54.253012 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:54.252630    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:32:55.251289 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:55.251251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:55.251469 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:55.251260    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:55.251469 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:55.251398    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:32:55.251588 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:55.251537    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:32:56.251632 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:56.251600    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:56.252069 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:56.251732    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:32:57.251562 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:57.251526    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:57.251562 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:57.251561    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:57.252111 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:57.251658    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:32:57.252111 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:57.251783    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:32:58.250943 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:58.250910    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:32:58.251128 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:58.251037    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:32:59.251357 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:59.251310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:32:59.251799 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:32:59.251324    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:32:59.251887 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:59.251851    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:32:59.252456 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:32:59.252425    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:00.251019 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:00.250991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:00.251125 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.251108    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:00.867262 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:00.867212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:00.867759 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:00.867289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:00.867759 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.867376    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:00.867759 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.867416    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:00.867759 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.867451    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs podName:9ab5b87b-81cc-4429-8147-c990db710694 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:16.867430616 +0000 UTC m=+33.117487602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs") pod "network-metrics-daemon-lx8xf" (UID: "9ab5b87b-81cc-4429-8147-c990db710694") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:00.867759 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.867472    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret podName:c75abded-e781-47c2-a233-e50561257d20 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:16.867462243 +0000 UTC m=+33.117519232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret") pod "global-pull-secret-syncer-f5ms2" (UID: "c75abded-e781-47c2-a233-e50561257d20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:00.967828 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:00.967793    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:00.968006 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.967963    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:00.968006 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.967985    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:00.968006 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.967995    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9l5c9 for pod openshift-network-diagnostics/network-check-target-q7wx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:00.968138 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:00.968050    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9 podName:a6fe2c5a-6be6-4a10-82af-f895ddcdfda2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:16.968035906 +0000 UTC m=+33.218092892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5c9" (UniqueName: "kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9") pod "network-check-target-q7wx2" (UID: "a6fe2c5a-6be6-4a10-82af-f895ddcdfda2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:01.251847 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:01.251814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:01.252010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:01.251819    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:01.252010 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:01.251965    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:01.252109 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:01.252002    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:02.251235 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:02.251200    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:02.251660 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:02.251327    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:03.251653 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:03.251617    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:03.252120 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:03.251626    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:03.252120 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:03.251742    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:03.252120 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:03.251828    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:04.253125 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:04.252769    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:04.253125 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:04.253053    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:04.438590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:04.438541    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kbmm9" event={"ID":"8014099c-4821-4606-bf04-5517c49877e7","Type":"ContainerStarted","Data":"1321a9353befe7f33df17a09a9468f0975f866f35523be169cc000f6040548ea"}
Jul 28 15:33:04.440312 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:04.440283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" event={"ID":"6807966616aea208bee6841100daf208","Type":"ContainerStarted","Data":"2a16a728c8b5842729913509c495ac6cd5eb921d7a366f79a50e1c961d6d9adb"}
Jul 28 15:33:04.448014 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:04.447991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"85c29dba62cf7076b263d19691a7759ca440135023ba033091ac2adaefcba1a3"}
Jul 28 15:33:04.448105 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:04.448050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"973528e3b65a573682326f09a50d3d402c79fbb9c33029307cace095227aff7a"}
Jul 28 15:33:04.478659 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:04.478607    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" podStartSLOduration=19.478591609 podStartE2EDuration="19.478591609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:04.478244847 +0000 UTC m=+20.728301853" watchObservedRunningTime="2026-07-28 15:33:04.478591609 +0000 UTC m=+20.728648648"
Jul 28 15:33:04.478773 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:04.478684    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kbmm9" podStartSLOduration=2.099479874 podStartE2EDuration="20.47867889s" totalImagesPullingTime="18.379199016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:04.460963711 +0000 UTC m=+20.711020715" watchObservedRunningTime="2026-07-28 15:33:04.47867889 +0000 UTC m=+20.728735894"
Jul 28 15:33:05.251180 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.250993    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:05.251355 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.250996    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:05.251355 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:05.251283    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:05.251477 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:05.251388    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:05.451912 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.451875    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7dpj" event={"ID":"d0d40f3e-c36b-4def-acaa-dcf458c09666","Type":"ContainerStarted","Data":"131a4e995f4f6dd73c4a34b3c53f408480acce20dac355ba3f12fc740f26d54c"}
Jul 28 15:33:05.453630 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.453596    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9cqsh" event={"ID":"f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1","Type":"ContainerStarted","Data":"b42b8b6980a3b5a227828b16aae94103746485a5c54d048a30b753a131453b86"}
Jul 28 15:33:05.456642 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.456624    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:33:05.457012 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.456992    2575 generic.go:356] "Generic (PLEG): container finished" podID="74285376-f5c6-4e10-a103-9fd61945f750" containerID="85c29dba62cf7076b263d19691a7759ca440135023ba033091ac2adaefcba1a3" exitCode=1
Jul 28 15:33:05.457084 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.457065    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerDied","Data":"85c29dba62cf7076b263d19691a7759ca440135023ba033091ac2adaefcba1a3"}
Jul 28 15:33:05.457143 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.457129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"c00f4058b9c066ca2e43310aa6ff0e325fff5c25c504d299ff82d85169c90943"}
Jul 28 15:33:05.457208 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.457200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"97c53fcc0e8e7030f12b7372e592c1368fffc39a53089aa4d28f114eef9373b9"}
Jul 28 15:33:05.457292 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.457249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"dc60c87d58d7ed41466dd3f42c0e283a40337737d5fd5cef678405f3bb26c871"}
Jul 28 15:33:05.457343 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.457303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"1c435717e5709e323e1522455d633b85bb8a35984cea99bb8c614d8074ac399f"}
Jul 28 15:33:05.458616 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.458583    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rr2gh" event={"ID":"30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882","Type":"ContainerStarted","Data":"a8f6b44537c39f98e488f863b06eaafca62ca3d8f2230d32a7a327725bdfa000"}
Jul 28 15:33:05.460132 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.460107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8v5vg" event={"ID":"5c0a5568-31d1-4001-b029-47265e985078","Type":"ContainerStarted","Data":"030bc129d21e11731c99b5fb2fc48426bc979ccb4aa7ceeabfef84d82336c6e1"}
Jul 28 15:33:05.461576 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.461557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2" event={"ID":"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f","Type":"ContainerStarted","Data":"fd8d1e0298ae5229b97a6d379bc8acbef870174f03dccf6d41982bbc1c08eb4d"}
Jul 28 15:33:05.463137 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.463113    2575 generic.go:356] "Generic (PLEG): container finished" podID="38e5b545-9292-4266-bdc0-63ba5e9c70e3" containerID="40fbef45dcc6b2ce774c8e31bf6a8686cb3644646cfd5a6684f911beb81c5819" exitCode=0
Jul 28 15:33:05.463245 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.463192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerDied","Data":"40fbef45dcc6b2ce774c8e31bf6a8686cb3644646cfd5a6684f911beb81c5819"}
Jul 28 15:33:05.464916 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.464842    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d9f3368b81aaba73b18a11f9ce1769e" containerID="afaea5889004152068894846a26b661668889cb6f6f2bccb3565c89867267038" exitCode=0
Jul 28 15:33:05.464916 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.464867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" event={"ID":"3d9f3368b81aaba73b18a11f9ce1769e","Type":"ContainerDied","Data":"afaea5889004152068894846a26b661668889cb6f6f2bccb3565c89867267038"}
Jul 28 15:33:05.490216 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.490196    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:33:05.490941 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.490925    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:33:05.522892 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.522846    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s7dpj" podStartSLOduration=2.97917954 podStartE2EDuration="21.522831599s" totalImagesPullingTime="18.543652059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:05.480640335 +0000 UTC m=+21.730697341" watchObservedRunningTime="2026-07-28 15:33:05.522831599 +0000 UTC m=+21.772888641"
Jul 28 15:33:05.608203 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.608140    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9cqsh" podStartSLOduration=2.605075497 podStartE2EDuration="21.60812511s" totalImagesPullingTime="19.003049613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:05.605800521 +0000 UTC m=+21.855857517" watchObservedRunningTime="2026-07-28 15:33:05.60812511 +0000 UTC m=+21.858182114"
Jul 28 15:33:05.608552 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.608531    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8v5vg" podStartSLOduration=2.9674507500000002 podStartE2EDuration="21.608522845s" totalImagesPullingTime="18.641072095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:05.553582401 +0000 UTC m=+21.803639405" watchObservedRunningTime="2026-07-28 15:33:05.608522845 +0000 UTC m=+21.858579850"
Jul 28 15:33:05.657737 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.657576    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rr2gh" podStartSLOduration=3.094588299 podStartE2EDuration="21.657557384s" totalImagesPullingTime="18.562969085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:05.657505468 +0000 UTC m=+21.907562471" watchObservedRunningTime="2026-07-28 15:33:05.657557384 +0000 UTC m=+21.907614394"
Jul 28 15:33:05.909616 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:05.909588    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 15:33:06.186369 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.186201    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T15:33:05.909611415Z","UUID":"dec3523f-423f-4179-988f-6e2f85c72a87","Handler":null,"Name":"","Endpoint":""}
Jul 28 15:33:06.187860 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.187831    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 15:33:06.187984 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.187879    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 15:33:06.251671 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.251644    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:06.251814 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:06.251761    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:06.469070 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.469036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" event={"ID":"3d9f3368b81aaba73b18a11f9ce1769e","Type":"ContainerStarted","Data":"ab0c5d140c25c9767644faf233b973da314dd655eba8a703ea637cb9f57d7fcd"}
Jul 28 15:33:06.470595 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.470565    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zvqkp" event={"ID":"bf75d18f-4f83-4f4d-8dc5-5f8e3e7987d4","Type":"ContainerStarted","Data":"32f1601a40927033e70de48af2fdbb29feb859562f75b37fe91a19d16ac1fb72"}
Jul 28 15:33:06.472637 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.472591    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2" event={"ID":"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f","Type":"ContainerStarted","Data":"f9504ceaf456115cbf08a57608e6eef70e1b88f64f8c54e9196f6c34f2fbe665"}
Jul 28 15:33:06.472792 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.472770    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:33:06.473281 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.473258    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s7dpj"
Jul 28 15:33:06.500259 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:06.500202    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" podStartSLOduration=21.500182482 podStartE2EDuration="21.500182482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:06.500066807 +0000 UTC m=+22.750123834" watchObservedRunningTime="2026-07-28 15:33:06.500182482 +0000 UTC m=+22.750239481"
Jul 28 15:33:07.251655 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:07.251625    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:07.251767 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:07.251657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:07.251767 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:07.251739    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:07.251893 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:07.251875    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:07.477565 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:07.477533    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2" event={"ID":"45e0a2a1-1e0c-4bdd-9aad-97e624f8de6f","Type":"ContainerStarted","Data":"8e41414fe884238fd8c844ae1136a94acade46a6b9b40fe557e72427969c90ba"}
Jul 28 15:33:07.480917 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:07.480897    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:33:07.481388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:07.481358    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"0b17df36ef9a8896d598bf0a319419449d0d0d8b40af8f45a1c5d68561b8c331"}
Jul 28 15:33:07.500868 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:07.500823    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zvqkp" podStartSLOduration=4.883305547 podStartE2EDuration="23.500812067s" totalImagesPullingTime="18.61750652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:06.55205746 +0000 UTC m=+22.802114466" watchObservedRunningTime="2026-07-28 15:33:07.500812067 +0000 UTC m=+23.750869072"
Jul 28 15:33:07.501323 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:07.501295    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq2x2" podStartSLOduration=2.754652991 podStartE2EDuration="23.501290217s" totalImagesPullingTime="20.746637226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:07.500605765 +0000 UTC m=+23.750662771" watchObservedRunningTime="2026-07-28 15:33:07.501290217 +0000 UTC m=+23.751347222"
Jul 28 15:33:08.251405 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:08.251380    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:08.251575 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:08.251490    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:09.251508 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:09.251425    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:09.251508 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:09.251468    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:09.251993 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:09.251540    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:09.251993 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:09.251683    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:10.251640 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:10.251463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:10.252273 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:10.251742    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:10.492446 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:10.492421    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:33:10.492801 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:10.492769    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"539ac3736267336730a9a5996722fe9df9b3771b2cae29edccb173f263a98473"}
Jul 28 15:33:10.493142 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:10.493061    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:33:10.493279 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:10.493192    2575 scope.go:117] "RemoveContainer" containerID="85c29dba62cf7076b263d19691a7759ca440135023ba033091ac2adaefcba1a3"
Jul 28 15:33:10.494730 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:10.494701    2575 generic.go:356] "Generic (PLEG): container finished" podID="38e5b545-9292-4266-bdc0-63ba5e9c70e3" containerID="87a18211c8bcb87e861c05076fb92c53fd85d5b24627a766f52302f5b053e1fd" exitCode=0
Jul 28 15:33:10.494802 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:10.494736    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerDied","Data":"87a18211c8bcb87e861c05076fb92c53fd85d5b24627a766f52302f5b053e1fd"}
Jul 28 15:33:10.511448 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:10.511430    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:33:11.250875 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.250845    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:11.250875 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.250874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:11.251102 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:11.250983    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:11.251102 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:11.251084    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:11.500687 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.500665    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:33:11.501118 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.501019    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" event={"ID":"74285376-f5c6-4e10-a103-9fd61945f750","Type":"ContainerStarted","Data":"9abbba06e557c86f4929a89d1e937a29336931c414bf2dcd183120a9afc4ce7c"}
Jul 28 15:33:11.501323 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.501305    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:33:11.501413 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.501376    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:33:11.503372 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.503351    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerStarted","Data":"26c7f5f6bf2a267a52d144ebf4402e824f577673a66a050788c3fd8938fa05c0"}
Jul 28 15:33:11.521445 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.521423    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:33:11.535995 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.535957    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh" podStartSLOduration=8.930729202 podStartE2EDuration="27.535945691s" totalImagesPullingTime="18.605216489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:11.534200661 +0000 UTC m=+27.784257666" watchObservedRunningTime="2026-07-28 15:33:11.535945691 +0000 UTC m=+27.786002697"
Jul 28 15:33:11.707426 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.707395    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lx8xf"]
Jul 28 15:33:11.707587 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.707515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:11.707644 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:11.707616    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:11.711076 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.711051    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f5ms2"]
Jul 28 15:33:11.711221 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.711149    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:11.711298 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:11.711257    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:11.711773 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.711751    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q7wx2"]
Jul 28 15:33:11.711857 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:11.711844    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:11.712000 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:11.711914    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:12.507495 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:12.507304    2575 generic.go:356] "Generic (PLEG): container finished" podID="38e5b545-9292-4266-bdc0-63ba5e9c70e3" containerID="26c7f5f6bf2a267a52d144ebf4402e824f577673a66a050788c3fd8938fa05c0" exitCode=0
Jul 28 15:33:12.507495 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:12.507389    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerDied","Data":"26c7f5f6bf2a267a52d144ebf4402e824f577673a66a050788c3fd8938fa05c0"}
Jul 28 15:33:13.251712 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:13.251676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:13.251860 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:13.251721    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:13.251860 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:13.251689    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:13.251860 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:13.251805    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:13.252011 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:13.251869    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:13.252011 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:13.251964    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:14.514281 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:14.514248    2575 generic.go:356] "Generic (PLEG): container finished" podID="38e5b545-9292-4266-bdc0-63ba5e9c70e3" containerID="0256e57afc2855f18da5eb01c8373daaef40b19b75d956c165c5a269de69afca" exitCode=0
Jul 28 15:33:14.514698 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:14.514287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerDied","Data":"0256e57afc2855f18da5eb01c8373daaef40b19b75d956c165c5a269de69afca"}
Jul 28 15:33:15.251300 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:15.251264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:15.251476 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:15.251264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:15.251476 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:15.251404    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q7wx2" podUID="a6fe2c5a-6be6-4a10-82af-f895ddcdfda2"
Jul 28 15:33:15.251476 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:15.251414    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:15.251641 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:15.251512    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lx8xf" podUID="9ab5b87b-81cc-4429-8147-c990db710694"
Jul 28 15:33:15.251641 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:15.251607    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5ms2" podUID="c75abded-e781-47c2-a233-e50561257d20"
Jul 28 15:33:16.888353 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:16.888320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:16.888900 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:16.888373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:16.888900 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:16.888471    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:16.888900 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:16.888484    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:16.888900 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:16.888522    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret podName:c75abded-e781-47c2-a233-e50561257d20 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:48.888509536 +0000 UTC m=+65.138566520 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret") pod "global-pull-secret-syncer-f5ms2" (UID: "c75abded-e781-47c2-a233-e50561257d20") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:16.888900 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:16.888583    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs podName:9ab5b87b-81cc-4429-8147-c990db710694 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:48.888533086 +0000 UTC m=+65.138590069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs") pod "network-metrics-daemon-lx8xf" (UID: "9ab5b87b-81cc-4429-8147-c990db710694") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:16.989497 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:16.989455    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:16.989634 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:16.989618    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:16.989674 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:16.989643    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:16.989674 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:16.989653    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9l5c9 for pod openshift-network-diagnostics/network-check-target-q7wx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:16.989738 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:16.989708    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9 podName:a6fe2c5a-6be6-4a10-82af-f895ddcdfda2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:48.989689434 +0000 UTC m=+65.239746420 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l5c9" (UniqueName: "kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9") pod "network-check-target-q7wx2" (UID: "a6fe2c5a-6be6-4a10-82af-f895ddcdfda2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:17.060607 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.060578    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeReady"
Jul 28 15:33:17.060771 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.060733    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 15:33:17.193904 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.193875    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f9ccb895b-vld76"]
Jul 28 15:33:17.195964 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.195948    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.202063 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.202038    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 15:33:17.207212 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.207190    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 15:33:17.231784 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.231758    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 15:33:17.232193 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.232154    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xl2f2\""
Jul 28 15:33:17.251838 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.251811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:17.251976 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.251850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:17.251976 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.251918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:17.274767 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.274742    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 15:33:17.291970 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.291939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-bound-sa-token\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.292111 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.291984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-trusted-ca\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.292111 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.292071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.292111 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.292103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-kube-api-access-qk77w\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.292286 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.292133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-certificates\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.292286 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.292196    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-image-registry-private-configuration\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.292286 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.292230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cab2757c-3cb4-4878-924f-776e26a1bb7b-ca-trust-extracted\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.292286 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.292252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-installation-pull-secrets\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.309732 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.309700    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 15:33:17.311099 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.311065    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f9ccb895b-vld76"]
Jul 28 15:33:17.311793 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.311772    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mznsm"]
Jul 28 15:33:17.312583 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.312562    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-25sjx\""
Jul 28 15:33:17.312975 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.312955    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2697\""
Jul 28 15:33:17.313808 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.313788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.363879 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.363848    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 15:33:17.363879 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.363873    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 15:33:17.364138 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.363849    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 15:33:17.364138 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.363848    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 15:33:17.390802 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.390771    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkn7m\""
Jul 28 15:33:17.392695 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.392668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cab2757c-3cb4-4878-924f-776e26a1bb7b-ca-trust-extracted\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.392814 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.392710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d0c8519-a071-4afa-a5ab-5bc282690423-tmp-dir\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.392814 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.392744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-installation-pull-secrets\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.392814 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.392769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.392972 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.392868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-bound-sa-token\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.392972 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.392911    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0c8519-a071-4afa-a5ab-5bc282690423-config-volume\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.392972 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.392938    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-trusted-ca\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.393119 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.393000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.393119 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.393026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-kube-api-access-qk77w\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.393119 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.393058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-certificates\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.393119 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.393090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-image-registry-private-configuration\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.393119 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.393097    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cab2757c-3cb4-4878-924f-776e26a1bb7b-ca-trust-extracted\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.393389 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.393121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xbm\" (UniqueName: \"kubernetes.io/projected/9d0c8519-a071-4afa-a5ab-5bc282690423-kube-api-access-86xbm\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.393389 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.393145    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:17.393389 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.393179    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9ccb895b-vld76: secret "image-registry-tls" not found
Jul 28 15:33:17.393389 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.393241    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls podName:cab2757c-3cb4-4878-924f-776e26a1bb7b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:17.893219373 +0000 UTC m=+34.143276375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls") pod "image-registry-6f9ccb895b-vld76" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b") : secret "image-registry-tls" not found
Jul 28 15:33:17.393669 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.393648    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-certificates\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.397602 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.397527    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mznsm"]
Jul 28 15:33:17.397942 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.397919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-image-registry-private-configuration\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.397942 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.397936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-installation-pull-secrets\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.398337 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.398317    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sqzxh"]
Jul 28 15:33:17.400372 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.400350    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sqzxh"
Jul 28 15:33:17.404248 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.404003    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 15:33:17.406562 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.406543    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 15:33:17.406667 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.406562    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lf5lf\""
Jul 28 15:33:17.406861 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.406845    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 15:33:17.408553 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.408536    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 15:33:17.413860 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.413833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-trusted-ca\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.413973 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.413943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-bound-sa-token\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.417766 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.417745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-kube-api-access-qk77w\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.424605 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.424583    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sqzxh"]
Jul 28 15:33:17.494283 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.494192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0c8519-a071-4afa-a5ab-5bc282690423-config-volume\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.494283 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.494245    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlzr\" (UniqueName: \"kubernetes.io/projected/c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a-kube-api-access-ltlzr\") pod \"ingress-canary-sqzxh\" (UID: \"c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a\") " pod="openshift-ingress-canary/ingress-canary-sqzxh"
Jul 28 15:33:17.494283 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.494276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a-cert\") pod \"ingress-canary-sqzxh\" (UID: \"c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a\") " pod="openshift-ingress-canary/ingress-canary-sqzxh"
Jul 28 15:33:17.494562 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.494350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xbm\" (UniqueName: \"kubernetes.io/projected/9d0c8519-a071-4afa-a5ab-5bc282690423-kube-api-access-86xbm\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.494562 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.494378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d0c8519-a071-4afa-a5ab-5bc282690423-tmp-dir\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.494562 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.494407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.494562 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.494545    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:17.494750 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.494627    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls podName:9d0c8519-a071-4afa-a5ab-5bc282690423 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:17.994607855 +0000 UTC m=+34.244664841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls") pod "dns-default-mznsm" (UID: "9d0c8519-a071-4afa-a5ab-5bc282690423") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:17.494800 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.494769    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d0c8519-a071-4afa-a5ab-5bc282690423-tmp-dir\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.494916 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.494885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0c8519-a071-4afa-a5ab-5bc282690423-config-volume\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.507298 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.507268    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xbm\" (UniqueName: \"kubernetes.io/projected/9d0c8519-a071-4afa-a5ab-5bc282690423-kube-api-access-86xbm\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.595202 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.595149    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlzr\" (UniqueName: \"kubernetes.io/projected/c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a-kube-api-access-ltlzr\") pod \"ingress-canary-sqzxh\" (UID: \"c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a\") " pod="openshift-ingress-canary/ingress-canary-sqzxh"
Jul 28 15:33:17.595387 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.595214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a-cert\") pod \"ingress-canary-sqzxh\" (UID: \"c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a\") " pod="openshift-ingress-canary/ingress-canary-sqzxh"
Jul 28 15:33:17.597803 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.597777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a-cert\") pod \"ingress-canary-sqzxh\" (UID: \"c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a\") " pod="openshift-ingress-canary/ingress-canary-sqzxh"
Jul 28 15:33:17.605262 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.605233    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlzr\" (UniqueName: \"kubernetes.io/projected/c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a-kube-api-access-ltlzr\") pod \"ingress-canary-sqzxh\" (UID: \"c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a\") " pod="openshift-ingress-canary/ingress-canary-sqzxh"
Jul 28 15:33:17.727182 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.726987    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sqzxh"
Jul 28 15:33:17.897516 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.897425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:17.897959 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.897608    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:17.897959 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.897627    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9ccb895b-vld76: secret "image-registry-tls" not found
Jul 28 15:33:17.897959 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.897680    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls podName:cab2757c-3cb4-4878-924f-776e26a1bb7b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:18.897665365 +0000 UTC m=+35.147722353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls") pod "image-registry-6f9ccb895b-vld76" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b") : secret "image-registry-tls" not found
Jul 28 15:33:17.998722 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:17.998682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:17.998898 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.998818    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:17.998898 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:17.998887    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls podName:9d0c8519-a071-4afa-a5ab-5bc282690423 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:18.99886674 +0000 UTC m=+35.248923743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls") pod "dns-default-mznsm" (UID: "9d0c8519-a071-4afa-a5ab-5bc282690423") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:18.614610 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:18.614566    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sqzxh"]
Jul 28 15:33:18.617239 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:33:18.617215    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c0ca25_e742_4a8c_b5cd_1e9f36cbea7a.slice/crio-3b6e7ee16cc5f2f01bb61c62bffa8d7e3a8d9d8a297bb8aae4b801d04ce18237 WatchSource:0}: Error finding container 3b6e7ee16cc5f2f01bb61c62bffa8d7e3a8d9d8a297bb8aae4b801d04ce18237: Status 404 returned error can't find the container with id 3b6e7ee16cc5f2f01bb61c62bffa8d7e3a8d9d8a297bb8aae4b801d04ce18237
Jul 28 15:33:18.906896 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:18.906828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:18.907531 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:18.906978    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:18.907531 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:18.906997    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9ccb895b-vld76: secret "image-registry-tls" not found
Jul 28 15:33:18.907531 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:18.907048    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls podName:cab2757c-3cb4-4878-924f-776e26a1bb7b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:20.907034501 +0000 UTC m=+37.157091484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls") pod "image-registry-6f9ccb895b-vld76" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b") : secret "image-registry-tls" not found
Jul 28 15:33:19.007575 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:19.007537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:19.007739 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:19.007701    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:19.007800 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:19.007774    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls podName:9d0c8519-a071-4afa-a5ab-5bc282690423 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:21.007752685 +0000 UTC m=+37.257809670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls") pod "dns-default-mznsm" (UID: "9d0c8519-a071-4afa-a5ab-5bc282690423") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:19.529832 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:19.529793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sqzxh" event={"ID":"c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a","Type":"ContainerStarted","Data":"3b6e7ee16cc5f2f01bb61c62bffa8d7e3a8d9d8a297bb8aae4b801d04ce18237"}
Jul 28 15:33:19.532214 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:19.532185    2575 generic.go:356] "Generic (PLEG): container finished" podID="38e5b545-9292-4266-bdc0-63ba5e9c70e3" containerID="299d587dc82eaa9cc7166999cb95ac8af709102a0a3b1465f89a45dc0e1ecbb6" exitCode=0
Jul 28 15:33:19.532316 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:19.532229    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerDied","Data":"299d587dc82eaa9cc7166999cb95ac8af709102a0a3b1465f89a45dc0e1ecbb6"}
Jul 28 15:33:20.537966 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:20.537794    2575 generic.go:356] "Generic (PLEG): container finished" podID="38e5b545-9292-4266-bdc0-63ba5e9c70e3" containerID="2b1cdb27c6ea04a1af16572be7183711e0fc12e2a887d9edada2430bc98a7fca" exitCode=0
Jul 28 15:33:20.537966 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:20.537875    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerDied","Data":"2b1cdb27c6ea04a1af16572be7183711e0fc12e2a887d9edada2430bc98a7fca"}
Jul 28 15:33:20.923896 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:20.923871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:20.924057 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:20.924037    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:20.924097 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:20.924059    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9ccb895b-vld76: secret "image-registry-tls" not found
Jul 28 15:33:20.924134 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:20.924123    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls podName:cab2757c-3cb4-4878-924f-776e26a1bb7b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:24.924107064 +0000 UTC m=+41.174164047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls") pod "image-registry-6f9ccb895b-vld76" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b") : secret "image-registry-tls" not found
Jul 28 15:33:21.024512 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:21.024469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:21.024678 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:21.024616    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:21.024753 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:21.024681    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls podName:9d0c8519-a071-4afa-a5ab-5bc282690423 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:25.024665942 +0000 UTC m=+41.274722925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls") pod "dns-default-mznsm" (UID: "9d0c8519-a071-4afa-a5ab-5bc282690423") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:21.544423 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:21.544382    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" event={"ID":"38e5b545-9292-4266-bdc0-63ba5e9c70e3","Type":"ContainerStarted","Data":"2cb2f24c5f1823a6c5c4490fa0d751449b7fa8d8235bf341d9a0b694dd8291ef"}
Jul 28 15:33:21.545820 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:21.545794    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sqzxh" event={"ID":"c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a","Type":"ContainerStarted","Data":"86bf7fb8e3e3c5cbb682658a24c7100b7bd8872c878396b545fb8e5d66f57463"}
Jul 28 15:33:21.578177 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:21.578116    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5zfcm" podStartSLOduration=8.971634817 podStartE2EDuration="37.578102498s" totalImagesPullingTime="28.606467681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:21.577285787 +0000 UTC m=+37.827342793" watchObservedRunningTime="2026-07-28 15:33:21.578102498 +0000 UTC m=+37.828159502"
Jul 28 15:33:21.600133 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:21.600091    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sqzxh" podStartSLOduration=2.381372838 podStartE2EDuration="4.600076309s" totalImagesPullingTime="2.218703471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:21.598954408 +0000 UTC m=+37.849011414" watchObservedRunningTime="2026-07-28 15:33:21.600076309 +0000 UTC m=+37.850133314"
Jul 28 15:33:24.953788 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:24.953747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:24.954231 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:24.953902    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:24.954231 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:24.953924    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9ccb895b-vld76: secret "image-registry-tls" not found
Jul 28 15:33:24.954231 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:24.953980    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls podName:cab2757c-3cb4-4878-924f-776e26a1bb7b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:32.953964416 +0000 UTC m=+49.204021399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls") pod "image-registry-6f9ccb895b-vld76" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b") : secret "image-registry-tls" not found
Jul 28 15:33:25.054445 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:25.054410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:25.054618 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:25.054561    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:25.054666 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:25.054622    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls podName:9d0c8519-a071-4afa-a5ab-5bc282690423 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:33.05460612 +0000 UTC m=+49.304663103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls") pod "dns-default-mznsm" (UID: "9d0c8519-a071-4afa-a5ab-5bc282690423") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:33.006761 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:33.006724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:33.007326 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:33.006910    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:33.007326 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:33.006937    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9ccb895b-vld76: secret "image-registry-tls" not found
Jul 28 15:33:33.007326 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:33.007019    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls podName:cab2757c-3cb4-4878-924f-776e26a1bb7b nodeName:}" failed. No retries permitted until 2026-07-28 15:33:49.006995197 +0000 UTC m=+65.257052186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls") pod "image-registry-6f9ccb895b-vld76" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b") : secret "image-registry-tls" not found
Jul 28 15:33:33.107930 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:33.107882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:33.108094 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:33.108052    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:33.108196 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:33.108126    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls podName:9d0c8519-a071-4afa-a5ab-5bc282690423 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:49.108106907 +0000 UTC m=+65.358163904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls") pod "dns-default-mznsm" (UID: "9d0c8519-a071-4afa-a5ab-5bc282690423") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:39.890517 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.890472    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-d229l"]
Jul 28 15:33:39.892739 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.892721    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:39.903339 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.903316    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc"]
Jul 28 15:33:39.905080 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.905064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc"
Jul 28 15:33:39.914386 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.914347    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 15:33:39.914527 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.914389    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 15:33:39.914597 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.914527    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 15:33:39.914597 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.914532    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j9qbd\""
Jul 28 15:33:39.914730 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.914715    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 15:33:39.915931 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.915902    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-842lk\""
Jul 28 15:33:39.916053 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.915967    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:33:39.916053 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.915988    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 15:33:39.927365 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.927343    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-d229l"]
Jul 28 15:33:39.929278 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.929255    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 15:33:39.931282 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.931261    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc"]
Jul 28 15:33:39.958071 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.958044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f021571-3315-4bf9-889b-110d52285a46-service-ca-bundle\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:39.958071 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.958072    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f021571-3315-4bf9-889b-110d52285a46-snapshots\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:39.958250 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.958091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f021571-3315-4bf9-889b-110d52285a46-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:39.958250 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.958110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f021571-3315-4bf9-889b-110d52285a46-serving-cert\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:39.958250 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.958202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f021571-3315-4bf9-889b-110d52285a46-tmp\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:39.958250 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.958248    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq75q\" (UniqueName: \"kubernetes.io/projected/34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6-kube-api-access-nq75q\") pod \"volume-data-source-validator-549c6d54cf-r6wcc\" (UID: \"34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc"
Jul 28 15:33:39.958390 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:39.958268    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gdh\" (UniqueName: \"kubernetes.io/projected/1f021571-3315-4bf9-889b-110d52285a46-kube-api-access-f4gdh\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.059355 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.059314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f021571-3315-4bf9-889b-110d52285a46-service-ca-bundle\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.059355 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.059354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f021571-3315-4bf9-889b-110d52285a46-snapshots\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.059574 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.059374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f021571-3315-4bf9-889b-110d52285a46-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.059574 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.059403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f021571-3315-4bf9-889b-110d52285a46-serving-cert\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.059574 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.059452    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f021571-3315-4bf9-889b-110d52285a46-tmp\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.059574 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.059513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq75q\" (UniqueName: \"kubernetes.io/projected/34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6-kube-api-access-nq75q\") pod \"volume-data-source-validator-549c6d54cf-r6wcc\" (UID: \"34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc"
Jul 28 15:33:40.059750 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.059713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gdh\" (UniqueName: \"kubernetes.io/projected/1f021571-3315-4bf9-889b-110d52285a46-kube-api-access-f4gdh\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.060022 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.059988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f021571-3315-4bf9-889b-110d52285a46-tmp\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.060022 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.060001    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f021571-3315-4bf9-889b-110d52285a46-service-ca-bundle\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.060228 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.060040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f021571-3315-4bf9-889b-110d52285a46-snapshots\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.060408 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.060389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f021571-3315-4bf9-889b-110d52285a46-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.063221 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.063202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f021571-3315-4bf9-889b-110d52285a46-serving-cert\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.074707 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.074681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq75q\" (UniqueName: \"kubernetes.io/projected/34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6-kube-api-access-nq75q\") pod \"volume-data-source-validator-549c6d54cf-r6wcc\" (UID: \"34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc"
Jul 28 15:33:40.075006 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.074987    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gdh\" (UniqueName: \"kubernetes.io/projected/1f021571-3315-4bf9-889b-110d52285a46-kube-api-access-f4gdh\") pod \"insights-operator-5885fdb759-d229l\" (UID: \"1f021571-3315-4bf9-889b-110d52285a46\") " pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.201134 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.201106    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-d229l"
Jul 28 15:33:40.212635 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.212608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc"
Jul 28 15:33:40.368449 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.368420    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-d229l"]
Jul 28 15:33:40.371180 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:33:40.371142    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f021571_3315_4bf9_889b_110d52285a46.slice/crio-fd243125c42d338fdfb77dfebb9f23c21ed88d86ed511c00d8088009cbc85478 WatchSource:0}: Error finding container fd243125c42d338fdfb77dfebb9f23c21ed88d86ed511c00d8088009cbc85478: Status 404 returned error can't find the container with id fd243125c42d338fdfb77dfebb9f23c21ed88d86ed511c00d8088009cbc85478
Jul 28 15:33:40.374607 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.374585    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc"]
Jul 28 15:33:40.377122 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:33:40.377103    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e5b2fb_cd79_42ea_9e4d_ae5617cf20d6.slice/crio-b165941005f3c145612e32e38636cd61200b14d8335803f696a37f24e98168d7 WatchSource:0}: Error finding container b165941005f3c145612e32e38636cd61200b14d8335803f696a37f24e98168d7: Status 404 returned error can't find the container with id b165941005f3c145612e32e38636cd61200b14d8335803f696a37f24e98168d7
Jul 28 15:33:40.591479 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.591394    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc" event={"ID":"34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6","Type":"ContainerStarted","Data":"b165941005f3c145612e32e38636cd61200b14d8335803f696a37f24e98168d7"}
Jul 28 15:33:40.592556 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.592525    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-d229l" event={"ID":"1f021571-3315-4bf9-889b-110d52285a46","Type":"ContainerStarted","Data":"fd243125c42d338fdfb77dfebb9f23c21ed88d86ed511c00d8088009cbc85478"}
Jul 28 15:33:40.794320 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.794288    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ps655"]
Jul 28 15:33:40.796869 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.796850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.804824 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.804805    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 15:33:40.804939 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.804903    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-78wvn\""
Jul 28 15:33:40.806252 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.806236    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 15:33:40.806502 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.806481    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:33:40.806561 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.806479    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 15:33:40.811955 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.811935    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 15:33:40.824325 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.824282    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ps655"]
Jul 28 15:33:40.868446 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.868369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/687b4a32-0489-4bb5-9e72-47bab2e0702e-config\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.868588 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.868505    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txq5z\" (UniqueName: \"kubernetes.io/projected/687b4a32-0489-4bb5-9e72-47bab2e0702e-kube-api-access-txq5z\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.868588 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.868536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/687b4a32-0489-4bb5-9e72-47bab2e0702e-serving-cert\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.868588 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.868556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/687b4a32-0489-4bb5-9e72-47bab2e0702e-trusted-ca\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.969223 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.969188    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txq5z\" (UniqueName: \"kubernetes.io/projected/687b4a32-0489-4bb5-9e72-47bab2e0702e-kube-api-access-txq5z\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.969630 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.969234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/687b4a32-0489-4bb5-9e72-47bab2e0702e-serving-cert\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.969630 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.969346    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/687b4a32-0489-4bb5-9e72-47bab2e0702e-trusted-ca\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.969630 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.969530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/687b4a32-0489-4bb5-9e72-47bab2e0702e-config\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.970357 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.970325    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/687b4a32-0489-4bb5-9e72-47bab2e0702e-config\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.970506 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.970434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/687b4a32-0489-4bb5-9e72-47bab2e0702e-trusted-ca\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.972121 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.972086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/687b4a32-0489-4bb5-9e72-47bab2e0702e-serving-cert\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:40.980893 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:40.980866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txq5z\" (UniqueName: \"kubernetes.io/projected/687b4a32-0489-4bb5-9e72-47bab2e0702e-kube-api-access-txq5z\") pod \"console-operator-584dcb6b54-ps655\" (UID: \"687b4a32-0489-4bb5-9e72-47bab2e0702e\") " pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:41.105343 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:41.105306    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:41.474188 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:41.474147    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ps655"]
Jul 28 15:33:41.476155 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:33:41.476130    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687b4a32_0489_4bb5_9e72_47bab2e0702e.slice/crio-5a2926e6742a928964c30c99965a49ccb355e4cc82b345a212297492c709827f WatchSource:0}: Error finding container 5a2926e6742a928964c30c99965a49ccb355e4cc82b345a212297492c709827f: Status 404 returned error can't find the container with id 5a2926e6742a928964c30c99965a49ccb355e4cc82b345a212297492c709827f
Jul 28 15:33:41.596583 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:41.596505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc" event={"ID":"34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6","Type":"ContainerStarted","Data":"a57cd8d29d802f96f68a5ab655de6de952e495e4b7ec3636f255679a477867fb"}
Jul 28 15:33:41.597875 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:41.597843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" event={"ID":"687b4a32-0489-4bb5-9e72-47bab2e0702e","Type":"ContainerStarted","Data":"5a2926e6742a928964c30c99965a49ccb355e4cc82b345a212297492c709827f"}
Jul 28 15:33:41.616673 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:41.616633    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r6wcc" podStartSLOduration=1.638872012 podStartE2EDuration="2.616619253s" totalImagesPullingTime="977.747241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:41.615989565 +0000 UTC m=+57.866046571" watchObservedRunningTime="2026-07-28 15:33:41.616619253 +0000 UTC m=+57.866676236"
Jul 28 15:33:42.602103 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:42.602058    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-d229l" event={"ID":"1f021571-3315-4bf9-889b-110d52285a46","Type":"ContainerStarted","Data":"3a61f0531d6c3150ed60055c049e0ab01fd3d35e04046a2f8772732d3d004039"}
Jul 28 15:33:42.625328 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:42.625107    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-d229l" podStartSLOduration=1.555325842 podStartE2EDuration="3.625086964s" totalImagesPullingTime="2.069761122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:42.624967757 +0000 UTC m=+58.875024761" watchObservedRunningTime="2026-07-28 15:33:42.625086964 +0000 UTC m=+58.875143973"
Jul 28 15:33:43.523576 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.523534    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk7vh"
Jul 28 15:33:43.755778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.755737    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k99wm"]
Jul 28 15:33:43.758023 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.758002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.763527 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:43.763492    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Jul 28 15:33:43.763759 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:43.763724    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-k99wm\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d" pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.765558 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.765536    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 15:33:43.765682 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:43.765534    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-wgs6z\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wgs6z\"" type="*v1.Secret"
Jul 28 15:33:43.774559 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.774505    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k99wm"]
Jul 28 15:33:43.795753 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.795726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzbh\" (UniqueName: \"kubernetes.io/projected/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-api-access-2tzbh\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.795840 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.795820    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-data-volume\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.795886 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.795856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-crio-socket\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.795924 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.795906    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.795974 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.795961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.897110 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.897077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzbh\" (UniqueName: \"kubernetes.io/projected/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-api-access-2tzbh\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.897262 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.897176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-data-volume\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.897262 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.897197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-crio-socket\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.897262 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.897219    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.897262 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.897245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.897404 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.897325    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-crio-socket\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.897404 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:43.897331    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:43.897467 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:43.897451    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:44.397428683 +0000 UTC m=+60.647485684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:43.897570 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.897553    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-data-volume\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:43.908327 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:43.908301    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzbh\" (UniqueName: \"kubernetes.io/projected/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-api-access-2tzbh\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:44.401278 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:44.401242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:44.401525 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:44.401369    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:44.401525 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:44.401435    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.401419268 +0000 UTC m=+61.651476251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:44.608377 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:44.608350    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/0.log"
Jul 28 15:33:44.608552 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:44.608389    2575 generic.go:356] "Generic (PLEG): container finished" podID="687b4a32-0489-4bb5-9e72-47bab2e0702e" containerID="2a32b56d180b3c62ad79b7691a31c7b8446ed84d7b59ab955c47b87813d75f08" exitCode=255
Jul 28 15:33:44.608552 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:44.608440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" event={"ID":"687b4a32-0489-4bb5-9e72-47bab2e0702e","Type":"ContainerDied","Data":"2a32b56d180b3c62ad79b7691a31c7b8446ed84d7b59ab955c47b87813d75f08"}
Jul 28 15:33:44.608720 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:44.608700    2575 scope.go:117] "RemoveContainer" containerID="2a32b56d180b3c62ad79b7691a31c7b8446ed84d7b59ab955c47b87813d75f08"
Jul 28 15:33:44.898258 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:44.898205    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition
Jul 28 15:33:44.898664 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:44.898300    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.398283156 +0000 UTC m=+61.648340139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : failed to sync secret cache: timed out waiting for the condition
Jul 28 15:33:45.145652 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.145619    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wgs6z\""
Jul 28 15:33:45.256282 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.256248    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 15:33:45.407912 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.407829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:45.407912 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.407869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:45.408089 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:45.407950    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:45.408089 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:45.407980    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:45.408089 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:45.408004    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:47.407989704 +0000 UTC m=+63.658046691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:45.408089 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:45.408034    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:46.408021938 +0000 UTC m=+62.658078921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:45.613334 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.613303    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/1.log"
Jul 28 15:33:45.613694 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.613680    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/0.log"
Jul 28 15:33:45.613797 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.613714    2575 generic.go:356] "Generic (PLEG): container finished" podID="687b4a32-0489-4bb5-9e72-47bab2e0702e" containerID="0e217121a0ff220effd0dfc6a4c6ebc1446955c11150bca68694d53d1217e3b7" exitCode=255
Jul 28 15:33:45.613832 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.613795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" event={"ID":"687b4a32-0489-4bb5-9e72-47bab2e0702e","Type":"ContainerDied","Data":"0e217121a0ff220effd0dfc6a4c6ebc1446955c11150bca68694d53d1217e3b7"}
Jul 28 15:33:45.613889 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.613878    2575 scope.go:117] "RemoveContainer" containerID="2a32b56d180b3c62ad79b7691a31c7b8446ed84d7b59ab955c47b87813d75f08"
Jul 28 15:33:45.614106 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.614085    2575 scope.go:117] "RemoveContainer" containerID="0e217121a0ff220effd0dfc6a4c6ebc1446955c11150bca68694d53d1217e3b7"
Jul 28 15:33:45.614307 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:45.614286    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ps655_openshift-console-operator(687b4a32-0489-4bb5-9e72-47bab2e0702e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" podUID="687b4a32-0489-4bb5-9e72-47bab2e0702e"
Jul 28 15:33:45.963664 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:45.963634    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rr2gh_30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882/dns-node-resolver/0.log"
Jul 28 15:33:46.416738 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:46.416653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:46.416886 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:46.416787    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:46.416886 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:46.416845    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:48.41683165 +0000 UTC m=+64.666888633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:46.563992 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:46.563967    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8v5vg_5c0a5568-31d1-4001-b029-47265e985078/node-ca/0.log"
Jul 28 15:33:46.617867 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:46.617841    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/1.log"
Jul 28 15:33:46.618194 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:46.618157    2575 scope.go:117] "RemoveContainer" containerID="0e217121a0ff220effd0dfc6a4c6ebc1446955c11150bca68694d53d1217e3b7"
Jul 28 15:33:46.618346 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:46.618327    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ps655_openshift-console-operator(687b4a32-0489-4bb5-9e72-47bab2e0702e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" podUID="687b4a32-0489-4bb5-9e72-47bab2e0702e"
Jul 28 15:33:47.425219 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:47.425153    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:47.425633 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:47.425289    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:47.425633 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:47.425357    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:51.425339048 +0000 UTC m=+67.675396031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:47.563726 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:47.563693    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sqzxh_c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a/serve-healthcheck-canary/0.log"
Jul 28 15:33:48.431901 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:48.431867    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:48.432275 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:48.432022    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:48.432275 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:48.432086    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:52.432070692 +0000 UTC m=+68.682127675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:48.936637 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:48.936595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:33:48.936804 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:48.936649    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:48.939485 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:48.939464    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 15:33:48.939592 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:48.939577    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 15:33:48.946988 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:48.946968    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 15:33:48.947052 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:48.947026    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs podName:9ab5b87b-81cc-4429-8147-c990db710694 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:52.947010485 +0000 UTC m=+129.197067468 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs") pod "network-metrics-daemon-lx8xf" (UID: "9ab5b87b-81cc-4429-8147-c990db710694") : secret "metrics-daemon-secret" not found
Jul 28 15:33:48.949050 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:48.949032    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c75abded-e781-47c2-a233-e50561257d20-original-pull-secret\") pod \"global-pull-secret-syncer-f5ms2\" (UID: \"c75abded-e781-47c2-a233-e50561257d20\") " pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:49.037611 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.037586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:33:49.037753 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.037628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:49.037753 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:49.037726    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 15:33:49.037753 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:49.037742    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9ccb895b-vld76: secret "image-registry-tls" not found
Jul 28 15:33:49.037853 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:49.037785    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls podName:cab2757c-3cb4-4878-924f-776e26a1bb7b nodeName:}" failed. No retries permitted until 2026-07-28 15:34:21.037771947 +0000 UTC m=+97.287828929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls") pod "image-registry-6f9ccb895b-vld76" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b") : secret "image-registry-tls" not found
Jul 28 15:33:49.040696 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.040681    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 15:33:49.051691 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.051673    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 15:33:49.061304 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.061281    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l5c9\" (UniqueName: \"kubernetes.io/projected/a6fe2c5a-6be6-4a10-82af-f895ddcdfda2-kube-api-access-9l5c9\") pod \"network-check-target-q7wx2\" (UID: \"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2\") " pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:49.069206 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.069191    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5ms2"
Jul 28 15:33:49.078324 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.078309    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2697\""
Jul 28 15:33:49.086037 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.086022    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:49.138276 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.138205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:33:49.138481 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:49.138343    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:33:49.138481 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:49.138388    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls podName:9d0c8519-a071-4afa-a5ab-5bc282690423 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:21.13837445 +0000 UTC m=+97.388431436 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls") pod "dns-default-mznsm" (UID: "9d0c8519-a071-4afa-a5ab-5bc282690423") : secret "dns-default-metrics-tls" not found
Jul 28 15:33:49.231141 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.231113    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f5ms2"]
Jul 28 15:33:49.233540 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:33:49.233515    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75abded_e781_47c2_a233_e50561257d20.slice/crio-07cfd96f8a303ede49e7d62ec144007e979f930468fff5122961ae70124a99a9 WatchSource:0}: Error finding container 07cfd96f8a303ede49e7d62ec144007e979f930468fff5122961ae70124a99a9: Status 404 returned error can't find the container with id 07cfd96f8a303ede49e7d62ec144007e979f930468fff5122961ae70124a99a9
Jul 28 15:33:49.258362 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.258341    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q7wx2"]
Jul 28 15:33:49.260982 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:33:49.260933    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fe2c5a_6be6_4a10_82af_f895ddcdfda2.slice/crio-36fe2b0ae7d7df53a01fbcaa268db1c41b642fd2468f8b6da14bb3d7dcee74ab WatchSource:0}: Error finding container 36fe2b0ae7d7df53a01fbcaa268db1c41b642fd2468f8b6da14bb3d7dcee74ab: Status 404 returned error can't find the container with id 36fe2b0ae7d7df53a01fbcaa268db1c41b642fd2468f8b6da14bb3d7dcee74ab
Jul 28 15:33:49.626045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.625960    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q7wx2" event={"ID":"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2","Type":"ContainerStarted","Data":"36fe2b0ae7d7df53a01fbcaa268db1c41b642fd2468f8b6da14bb3d7dcee74ab"}
Jul 28 15:33:49.626867 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:49.626845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f5ms2" event={"ID":"c75abded-e781-47c2-a233-e50561257d20","Type":"ContainerStarted","Data":"07cfd96f8a303ede49e7d62ec144007e979f930468fff5122961ae70124a99a9"}
Jul 28 15:33:51.106429 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:51.106306    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:51.106429 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:51.106369    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:33:51.106984 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:51.106829    2575 scope.go:117] "RemoveContainer" containerID="0e217121a0ff220effd0dfc6a4c6ebc1446955c11150bca68694d53d1217e3b7"
Jul 28 15:33:51.107138 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:51.107068    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ps655_openshift-console-operator(687b4a32-0489-4bb5-9e72-47bab2e0702e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" podUID="687b4a32-0489-4bb5-9e72-47bab2e0702e"
Jul 28 15:33:51.459516 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:51.459478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:51.459702 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:51.459605    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:33:51.459702 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:51.459682    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:33:59.459661331 +0000 UTC m=+75.709718323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : configmap "kube-rbac-proxy" not found
Jul 28 15:33:52.466781 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:52.466738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:52.467237 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:52.466892    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:52.467237 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:33:52.466961    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:00.466944182 +0000 UTC m=+76.717001165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:33:53.259130 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.259094    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-nqwx8"]
Jul 28 15:33:53.261065 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.261039    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.264856 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.264832    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 15:33:53.265676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.265655    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-sqk6b\""
Jul 28 15:33:53.265777 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.265662    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 15:33:53.266599 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.266579    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 15:33:53.266713 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.266579    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 15:33:53.274561 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.274538    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-nqwx8"]
Jul 28 15:33:53.373384 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.373346    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj97k\" (UniqueName: \"kubernetes.io/projected/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-kube-api-access-sj97k\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.373384 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.373387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-signing-cabundle\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.373598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.373535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-signing-key\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.474391 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.474357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj97k\" (UniqueName: \"kubernetes.io/projected/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-kube-api-access-sj97k\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.474858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.474400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-signing-cabundle\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.474858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.474522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-signing-key\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.475182 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.475141    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-signing-cabundle\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.477184 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.477146    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-signing-key\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.486175 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.486133    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj97k\" (UniqueName: \"kubernetes.io/projected/bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6-kube-api-access-sj97k\") pod \"service-ca-658d7b7c78-nqwx8\" (UID: \"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.571456 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.571390    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8"
Jul 28 15:33:53.720817 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:53.720786    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-nqwx8"]
Jul 28 15:33:53.724596 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:33:53.724571    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd34f67f_f6ba_43bb_b8ea_c7cef0207ac6.slice/crio-29874f826c6d2b7e8466df2ccbcf64a609bf5f9a3f4136aa16c19cba31517c11 WatchSource:0}: Error finding container 29874f826c6d2b7e8466df2ccbcf64a609bf5f9a3f4136aa16c19cba31517c11: Status 404 returned error can't find the container with id 29874f826c6d2b7e8466df2ccbcf64a609bf5f9a3f4136aa16c19cba31517c11
Jul 28 15:33:54.643494 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:54.643447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8" event={"ID":"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6","Type":"ContainerStarted","Data":"29874f826c6d2b7e8466df2ccbcf64a609bf5f9a3f4136aa16c19cba31517c11"}
Jul 28 15:33:54.645147 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:54.645116    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q7wx2" event={"ID":"a6fe2c5a-6be6-4a10-82af-f895ddcdfda2","Type":"ContainerStarted","Data":"d455e214aec36c17f99488e07e10b9a9b32468d04a6459d81fceb4ce35ac3838"}
Jul 28 15:33:54.645313 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:54.645234    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:33:54.647068 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:54.647030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f5ms2" event={"ID":"c75abded-e781-47c2-a233-e50561257d20","Type":"ContainerStarted","Data":"c4d716b38992e89bd21e555cfe19b342e44e5eabe485f05671df539ae391625a"}
Jul 28 15:33:54.674649 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:54.674598    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q7wx2" podStartSLOduration=66.344384032 podStartE2EDuration="1m10.674578061s" totalImagesPullingTime="4.330194029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:54.673392955 +0000 UTC m=+70.923449974" watchObservedRunningTime="2026-07-28 15:33:54.674578061 +0000 UTC m=+70.924635069"
Jul 28 15:33:55.652025 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:55.651939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8" event={"ID":"bd34f67f-f6ba-43bb-b8ea-c7cef0207ac6","Type":"ContainerStarted","Data":"3b4a1833d392d2ef54c4205d354ee9230646611451c2e972ca5a2aa1797153a0"}
Jul 28 15:33:55.673130 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:55.673085    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-nqwx8" podStartSLOduration=1.069164229 podStartE2EDuration="2.67307046s" totalImagesPullingTime="1.603906231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:55.672523393 +0000 UTC m=+71.922580399" watchObservedRunningTime="2026-07-28 15:33:55.67307046 +0000 UTC m=+71.923127465"
Jul 28 15:33:55.673418 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:55.673398    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f5ms2" podStartSLOduration=67.312400102 podStartE2EDuration="1m11.673391995s" totalImagesPullingTime="4.360991893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:54.693353479 +0000 UTC m=+70.943410485" watchObservedRunningTime="2026-07-28 15:33:55.673391995 +0000 UTC m=+71.923449008"
Jul 28 15:33:59.524413 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:59.524357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:33:59.524906 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:33:59.524882    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:34:00.534242 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:00.534206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:34:00.534621 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:34:00.534349    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:00.534621 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:34:00.534402    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls podName:f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:16.534388057 +0000 UTC m=+92.784445040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99wm" (UID: "f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:03.251858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:03.251826    2575 scope.go:117] "RemoveContainer" containerID="0e217121a0ff220effd0dfc6a4c6ebc1446955c11150bca68694d53d1217e3b7"
Jul 28 15:34:03.682770 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:03.682694    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:34:03.683116 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:03.683100    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/1.log"
Jul 28 15:34:03.683206 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:03.683134    2575 generic.go:356] "Generic (PLEG): container finished" podID="687b4a32-0489-4bb5-9e72-47bab2e0702e" containerID="b230852436973572f9c2b58f83bf6b108470159438167c2b697a4fb129e5e694" exitCode=255
Jul 28 15:34:03.683206 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:03.683159    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" event={"ID":"687b4a32-0489-4bb5-9e72-47bab2e0702e","Type":"ContainerDied","Data":"b230852436973572f9c2b58f83bf6b108470159438167c2b697a4fb129e5e694"}
Jul 28 15:34:03.683299 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:03.683230    2575 scope.go:117] "RemoveContainer" containerID="0e217121a0ff220effd0dfc6a4c6ebc1446955c11150bca68694d53d1217e3b7"
Jul 28 15:34:03.683558 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:03.683543    2575 scope.go:117] "RemoveContainer" containerID="b230852436973572f9c2b58f83bf6b108470159438167c2b697a4fb129e5e694"
Jul 28 15:34:03.683726 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:34:03.683706    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-ps655_openshift-console-operator(687b4a32-0489-4bb5-9e72-47bab2e0702e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" podUID="687b4a32-0489-4bb5-9e72-47bab2e0702e"
Jul 28 15:34:04.687513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:04.687489    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:34:11.106268 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:11.106133    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:34:11.106268 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:11.106206    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:34:11.106753 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:11.106491    2575 scope.go:117] "RemoveContainer" containerID="b230852436973572f9c2b58f83bf6b108470159438167c2b697a4fb129e5e694"
Jul 28 15:34:11.106753 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:34:11.106642    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-ps655_openshift-console-operator(687b4a32-0489-4bb5-9e72-47bab2e0702e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" podUID="687b4a32-0489-4bb5-9e72-47bab2e0702e"
Jul 28 15:34:14.489360 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.489327    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"]
Jul 28 15:34:14.494443 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.494426    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.526716 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.526692    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 15:34:14.527982 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.527958    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 15:34:14.531913 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.531895    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 15:34:14.538590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.538571    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 15:34:14.578483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.578458    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"]
Jul 28 15:34:14.633847 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.633820    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"]
Jul 28 15:34:14.637491 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.637469    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"
Jul 28 15:34:14.641944 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.641925    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hkbqf\""
Jul 28 15:34:14.642130 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.642111    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 15:34:14.642573 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.642545    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 15:34:14.649320 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.649300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-tmp\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.649405 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.649352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf5cc\" (UniqueName: \"kubernetes.io/projected/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-kube-api-access-cf5cc\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.649475 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.649439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-klusterlet-config\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.656857 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.656831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"]
Jul 28 15:34:14.697218 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.697186    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"]
Jul 28 15:34:14.700639 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.700624    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"
Jul 28 15:34:14.706323 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.706241    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 15:34:14.706323 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.706261    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-f4h5r\""
Jul 28 15:34:14.716195 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.716176    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"]
Jul 28 15:34:14.750491 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.750430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-klusterlet-config\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.750607 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.750490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/988272ab-9de5-4cce-a773-c91dc24e3a3f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b9tl6\" (UID: \"988272ab-9de5-4cce-a773-c91dc24e3a3f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"
Jul 28 15:34:14.750607 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.750525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-tmp\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.750607 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.750579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5cc\" (UniqueName: \"kubernetes.io/projected/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-kube-api-access-cf5cc\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.750737 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.750629    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/988272ab-9de5-4cce-a773-c91dc24e3a3f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-b9tl6\" (UID: \"988272ab-9de5-4cce-a773-c91dc24e3a3f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"
Jul 28 15:34:14.750935 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.750919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-tmp\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.752838 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.752823    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-klusterlet-config\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.777079 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.777057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf5cc\" (UniqueName: \"kubernetes.io/projected/59ac0d0d-ff44-44d7-ade6-b885e24fdd42-kube-api-access-cf5cc\") pod \"klusterlet-addon-workmgr-5579fdf64-fkrnx\" (UID: \"59ac0d0d-ff44-44d7-ade6-b885e24fdd42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.802488 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.802463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:14.851607 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.851575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/988272ab-9de5-4cce-a773-c91dc24e3a3f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-b9tl6\" (UID: \"988272ab-9de5-4cce-a773-c91dc24e3a3f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"
Jul 28 15:34:14.851758 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.851737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/988272ab-9de5-4cce-a773-c91dc24e3a3f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b9tl6\" (UID: \"988272ab-9de5-4cce-a773-c91dc24e3a3f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"
Jul 28 15:34:14.851838 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.851792    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea384ab4-a0d6-4275-bb56-e2017933fdd0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-pv5nh\" (UID: \"ea384ab4-a0d6-4275-bb56-e2017933fdd0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"
Jul 28 15:34:14.852472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.852442    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/988272ab-9de5-4cce-a773-c91dc24e3a3f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-b9tl6\" (UID: \"988272ab-9de5-4cce-a773-c91dc24e3a3f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"
Jul 28 15:34:14.854413 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.854377    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/988272ab-9de5-4cce-a773-c91dc24e3a3f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b9tl6\" (UID: \"988272ab-9de5-4cce-a773-c91dc24e3a3f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"
Jul 28 15:34:14.935671 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.935639    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"]
Jul 28 15:34:14.938267 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:14.938236    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ac0d0d_ff44_44d7_ade6_b885e24fdd42.slice/crio-cc245e61c5a3a93f0af78baf3b8ac03d2e6da8e34498c3b340794fc13fba0020 WatchSource:0}: Error finding container cc245e61c5a3a93f0af78baf3b8ac03d2e6da8e34498c3b340794fc13fba0020: Status 404 returned error can't find the container with id cc245e61c5a3a93f0af78baf3b8ac03d2e6da8e34498c3b340794fc13fba0020
Jul 28 15:34:14.946241 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.946217    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"
Jul 28 15:34:14.952916 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.952897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea384ab4-a0d6-4275-bb56-e2017933fdd0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-pv5nh\" (UID: \"ea384ab4-a0d6-4275-bb56-e2017933fdd0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"
Jul 28 15:34:14.955270 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:14.955250    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea384ab4-a0d6-4275-bb56-e2017933fdd0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-pv5nh\" (UID: \"ea384ab4-a0d6-4275-bb56-e2017933fdd0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"
Jul 28 15:34:15.008781 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:15.008722    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"
Jul 28 15:34:15.072332 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:15.072292    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6"]
Jul 28 15:34:15.075738 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:15.075714    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988272ab_9de5_4cce_a773_c91dc24e3a3f.slice/crio-0dfae0999112e49ad889d464644330d55a7c8eb9de39045b6fb2ea14acfe59ef WatchSource:0}: Error finding container 0dfae0999112e49ad889d464644330d55a7c8eb9de39045b6fb2ea14acfe59ef: Status 404 returned error can't find the container with id 0dfae0999112e49ad889d464644330d55a7c8eb9de39045b6fb2ea14acfe59ef
Jul 28 15:34:15.130092 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:15.129966    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"]
Jul 28 15:34:15.132426 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:15.132403    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea384ab4_a0d6_4275_bb56_e2017933fdd0.slice/crio-e817ccc42a8f7a7a2efa2fde388b3b2330dbd94303eb2a2efb648ab64505cbe7 WatchSource:0}: Error finding container e817ccc42a8f7a7a2efa2fde388b3b2330dbd94303eb2a2efb648ab64505cbe7: Status 404 returned error can't find the container with id e817ccc42a8f7a7a2efa2fde388b3b2330dbd94303eb2a2efb648ab64505cbe7
Jul 28 15:34:15.724153 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:15.724112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh" event={"ID":"ea384ab4-a0d6-4275-bb56-e2017933fdd0","Type":"ContainerStarted","Data":"e817ccc42a8f7a7a2efa2fde388b3b2330dbd94303eb2a2efb648ab64505cbe7"}
Jul 28 15:34:15.726916 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:15.726871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx" event={"ID":"59ac0d0d-ff44-44d7-ade6-b885e24fdd42","Type":"ContainerStarted","Data":"cc245e61c5a3a93f0af78baf3b8ac03d2e6da8e34498c3b340794fc13fba0020"}
Jul 28 15:34:15.728432 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:15.728404    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6" event={"ID":"988272ab-9de5-4cce-a773-c91dc24e3a3f","Type":"ContainerStarted","Data":"0dfae0999112e49ad889d464644330d55a7c8eb9de39045b6fb2ea14acfe59ef"}
Jul 28 15:34:16.568370 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:16.568339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:34:16.571079 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:16.571044    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99wm\" (UID: \"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d\") " pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:34:16.782849 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:16.782462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k99wm"
Jul 28 15:34:17.518527 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:17.518499    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k99wm"]
Jul 28 15:34:17.520701 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:17.520671    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a0a4ce_da75_4e4e_a659_ff20f6f7dc1d.slice/crio-42eab10395bf1c2200e33fb0f7372459ddbdbf1eed1435df91ee5cb221495109 WatchSource:0}: Error finding container 42eab10395bf1c2200e33fb0f7372459ddbdbf1eed1435df91ee5cb221495109: Status 404 returned error can't find the container with id 42eab10395bf1c2200e33fb0f7372459ddbdbf1eed1435df91ee5cb221495109
Jul 28 15:34:17.737401 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:17.737361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh" event={"ID":"ea384ab4-a0d6-4275-bb56-e2017933fdd0","Type":"ContainerStarted","Data":"0ba7c0a7202a5b36fe11c07d3b5c9b5112f3429094959a3135cedde19d5e1239"}
Jul 28 15:34:17.737788 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:17.737760    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"
Jul 28 15:34:17.738895 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:17.738870    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k99wm" event={"ID":"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d","Type":"ContainerStarted","Data":"42eab10395bf1c2200e33fb0f7372459ddbdbf1eed1435df91ee5cb221495109"}
Jul 28 15:34:17.740700 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:17.740662    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6" event={"ID":"988272ab-9de5-4cce-a773-c91dc24e3a3f","Type":"ContainerStarted","Data":"32943949c96a4abfe6f21d9c3fde489be2a379dcc89f7b9127be0671acd5ca73"}
Jul 28 15:34:17.743579 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:17.743557    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh"
Jul 28 15:34:17.756712 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:17.756659    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pv5nh" podStartSLOduration=1.516693939 podStartE2EDuration="3.756641024s" totalImagesPullingTime="2.239947085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.755949236 +0000 UTC m=+94.006006256" watchObservedRunningTime="2026-07-28 15:34:17.756641024 +0000 UTC m=+94.006698031"
Jul 28 15:34:17.793903 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:17.793854    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b9tl6" podStartSLOduration=1.495169527 podStartE2EDuration="3.793838459s" totalImagesPullingTime="2.298668932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.793143645 +0000 UTC m=+94.043200652" watchObservedRunningTime="2026-07-28 15:34:17.793838459 +0000 UTC m=+94.043895464"
Jul 28 15:34:18.685713 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.685678    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-dqcch"]
Jul 28 15:34:18.687732 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.687709    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.692815 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.692704    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 15:34:18.692815 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.692748    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-z7z5m\""
Jul 28 15:34:18.692815 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.692766    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 15:34:18.692815 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.692790    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 15:34:18.692815 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.692796    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 15:34:18.693105 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.692901    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 15:34:18.706105 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.706084    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-dqcch"]
Jul 28 15:34:18.788665 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.788615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.788665 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.788657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czsb\" (UniqueName: \"kubernetes.io/projected/7b139057-3b86-4cd8-a749-c9185310e0dc-kube-api-access-2czsb\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.788857 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.788766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.788857 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.788792    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b139057-3b86-4cd8-a749-c9185310e0dc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.891089 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.890084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b139057-3b86-4cd8-a749-c9185310e0dc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.891089 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.890210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.891089 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.890248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czsb\" (UniqueName: \"kubernetes.io/projected/7b139057-3b86-4cd8-a749-c9185310e0dc-kube-api-access-2czsb\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.891089 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.890378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.891089 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:34:18.890514    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 15:34:18.891089 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:34:18.890587    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-tls podName:7b139057-3b86-4cd8-a749-c9185310e0dc nodeName:}" failed. No retries permitted until 2026-07-28 15:34:19.390567079 +0000 UTC m=+95.640624067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-dqcch" (UID: "7b139057-3b86-4cd8-a749-c9185310e0dc") : secret "prometheus-operator-tls" not found
Jul 28 15:34:18.892508 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.892462    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b139057-3b86-4cd8-a749-c9185310e0dc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.895043 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.895017    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:18.902177 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:18.902090    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czsb\" (UniqueName: \"kubernetes.io/projected/7b139057-3b86-4cd8-a749-c9185310e0dc-kube-api-access-2czsb\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:19.394853 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:19.394817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:19.397490 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:19.397464    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b139057-3b86-4cd8-a749-c9185310e0dc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-dqcch\" (UID: \"7b139057-3b86-4cd8-a749-c9185310e0dc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:19.596835 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:19.596789    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch"
Jul 28 15:34:19.738659 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:19.738613    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-dqcch"]
Jul 28 15:34:19.742126 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:19.742071    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b139057_3b86_4cd8_a749_c9185310e0dc.slice/crio-65169f773471277345d0e6da977c1b9c80616c56610e42b265e8f0e3f2fd0a39 WatchSource:0}: Error finding container 65169f773471277345d0e6da977c1b9c80616c56610e42b265e8f0e3f2fd0a39: Status 404 returned error can't find the container with id 65169f773471277345d0e6da977c1b9c80616c56610e42b265e8f0e3f2fd0a39
Jul 28 15:34:19.749869 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:19.749845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch" event={"ID":"7b139057-3b86-4cd8-a749-c9185310e0dc","Type":"ContainerStarted","Data":"65169f773471277345d0e6da977c1b9c80616c56610e42b265e8f0e3f2fd0a39"}
Jul 28 15:34:19.751561 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:19.751536    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx" event={"ID":"59ac0d0d-ff44-44d7-ade6-b885e24fdd42","Type":"ContainerStarted","Data":"7ca3d3fa4e14a4c7e03b640f44c90b2c1c81bbbcad9ee3a856be086762b0eb44"}
Jul 28 15:34:19.768769 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:19.768729    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx" podStartSLOduration=2.010981859 podStartE2EDuration="5.768714497s" totalImagesPullingTime="3.757732638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:19.767651108 +0000 UTC m=+96.017708113" watchObservedRunningTime="2026-07-28 15:34:19.768714497 +0000 UTC m=+96.018771480"
Jul 28 15:34:20.755502 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:20.755468    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:20.757451 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:20.757422    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5579fdf64-fkrnx"
Jul 28 15:34:21.108679 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:21.108590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:34:21.111407 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:21.111369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"image-registry-6f9ccb895b-vld76\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") " pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:34:21.209575 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:21.209540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:34:21.212153 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:21.212091    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d0c8519-a071-4afa-a5ab-5bc282690423-metrics-tls\") pod \"dns-default-mznsm\" (UID: \"9d0c8519-a071-4afa-a5ab-5bc282690423\") " pod="openshift-dns/dns-default-mznsm"
Jul 28 15:34:21.227289 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:21.227263    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkn7m\""
Jul 28 15:34:21.234255 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:21.234232    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mznsm"
Jul 28 15:34:21.409447 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:21.409363    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xl2f2\""
Jul 28 15:34:21.417053 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:21.417011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:34:23.327972 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.327942    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f9ccb895b-vld76"]
Jul 28 15:34:23.544071 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:23.543976    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab2757c_3cb4_4878_924f_776e26a1bb7b.slice/crio-bdf655a776c5b86e45249445783612b8575949f2bcd819bdf370e9ac91ced2cb WatchSource:0}: Error finding container bdf655a776c5b86e45249445783612b8575949f2bcd819bdf370e9ac91ced2cb: Status 404 returned error can't find the container with id bdf655a776c5b86e45249445783612b8575949f2bcd819bdf370e9ac91ced2cb
Jul 28 15:34:23.681779 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.681757    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mznsm"]
Jul 28 15:34:23.683661 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:23.683626    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0c8519_a071_4afa_a5ab_5bc282690423.slice/crio-0cf8ec80bed24b354f75d428b4c419ee78ddd636021165e8641dcd79bae35e62 WatchSource:0}: Error finding container 0cf8ec80bed24b354f75d428b4c419ee78ddd636021165e8641dcd79bae35e62: Status 404 returned error can't find the container with id 0cf8ec80bed24b354f75d428b4c419ee78ddd636021165e8641dcd79bae35e62
Jul 28 15:34:23.767750 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.767708    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch" event={"ID":"7b139057-3b86-4cd8-a749-c9185310e0dc","Type":"ContainerStarted","Data":"d1be25402033052d153023489d939e4dea6d27bef9abc9a8a260f2175d280606"}
Jul 28 15:34:23.767925 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.767778    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch" event={"ID":"7b139057-3b86-4cd8-a749-c9185310e0dc","Type":"ContainerStarted","Data":"6094e0043637e084800a651e25d83c2732b372ea4fc53b0c16bc728fa8b9104f"}
Jul 28 15:34:23.768844 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.768807    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mznsm" event={"ID":"9d0c8519-a071-4afa-a5ab-5bc282690423","Type":"ContainerStarted","Data":"0cf8ec80bed24b354f75d428b4c419ee78ddd636021165e8641dcd79bae35e62"}
Jul 28 15:34:23.770366 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.770339    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76" event={"ID":"cab2757c-3cb4-4878-924f-776e26a1bb7b","Type":"ContainerStarted","Data":"6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f"}
Jul 28 15:34:23.770462 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.770384    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76" event={"ID":"cab2757c-3cb4-4878-924f-776e26a1bb7b","Type":"ContainerStarted","Data":"bdf655a776c5b86e45249445783612b8575949f2bcd819bdf370e9ac91ced2cb"}
Jul 28 15:34:23.770462 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.770410    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:34:23.771740 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.771719    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k99wm" event={"ID":"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d","Type":"ContainerStarted","Data":"ec26542f369a201e7623fb86e2bc9910f82f292fb1ecbe9446835be22dcd22f1"}
Jul 28 15:34:23.786964 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.786909    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-dqcch" podStartSLOduration=1.987409625 podStartE2EDuration="5.78688862s" totalImagesPullingTime="3.799478995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:23.786869109 +0000 UTC m=+100.036926117" watchObservedRunningTime="2026-07-28 15:34:23.78688862 +0000 UTC m=+100.036945626"
Jul 28 15:34:23.812355 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:23.812247    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76" podStartSLOduration=99.812226571 podStartE2EDuration="1m39.812226571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:23.810841031 +0000 UTC m=+100.060898036" watchObservedRunningTime="2026-07-28 15:34:23.812226571 +0000 UTC m=+100.062283577"
Jul 28 15:34:24.253581 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:24.253549    2575 scope.go:117] "RemoveContainer" containerID="b230852436973572f9c2b58f83bf6b108470159438167c2b697a4fb129e5e694"
Jul 28 15:34:24.778839 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:24.778804    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:34:24.779308 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:24.778944    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" event={"ID":"687b4a32-0489-4bb5-9e72-47bab2e0702e","Type":"ContainerStarted","Data":"dd8342d6da91bc656e4c63d65eeebb4c5970beb5a80b63cc45434d4a58b0d645"}
Jul 28 15:34:24.779610 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:24.779565    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:34:24.804952 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:24.804885    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-ps655" podStartSLOduration=42.665378076 podStartE2EDuration="44.804866052s" totalImagesPullingTime="2.139487976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:24.802383848 +0000 UTC m=+101.052440854" watchObservedRunningTime="2026-07-28 15:34:24.804866052 +0000 UTC m=+101.054923058"
Jul 28 15:34:25.389637 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:25.389570    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-ps655"
Jul 28 15:34:25.654807 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:25.654719    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q7wx2"
Jul 28 15:34:26.094798 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.094763    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vspq2"]
Jul 28 15:34:26.157810 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.157779    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.160859 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.160791    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 15:34:26.160859 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.160845    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4lhqk\""
Jul 28 15:34:26.161045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.160907    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 15:34:26.161045 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.160939    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 15:34:26.261118 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b57bk\" (UniqueName: \"kubernetes.io/projected/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-kube-api-access-b57bk\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.261118 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-wtmp\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.261366 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261185    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-root\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.261366 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261221    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-metrics-client-ca\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.261366 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-sys\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.261366 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-textfile\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.261366 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-tls\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.261366 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261360    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.261613 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.261406    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-accelerators-collector-config\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.362725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.362637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.362725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.362692    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-accelerators-collector-config\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.362963 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.362820    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b57bk\" (UniqueName: \"kubernetes.io/projected/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-kube-api-access-b57bk\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.362963 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.362869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-wtmp\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.362963 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.362911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-root\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.362963 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.362953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-metrics-client-ca\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363148 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363094    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-wtmp\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363229 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-sys\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363229 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-root\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363229 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-textfile\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363368 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-tls\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363368 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-accelerators-collector-config\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363467 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-sys\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363467 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363456    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-metrics-client-ca\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.363576 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.363536    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-textfile\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.365759 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.365725    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.365992 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.365966    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-node-exporter-tls\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.371736 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.371712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b57bk\" (UniqueName: \"kubernetes.io/projected/facb0c31-45fd-47c4-8d9f-b9b86b8bcc62-kube-api-access-b57bk\") pod \"node-exporter-vspq2\" (UID: \"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62\") " pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:26.470459 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:26.470424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vspq2"
Jul 28 15:34:27.160032 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.159998    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:34:27.181293 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.181249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.182861 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.182805    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:34:27.184266 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184242    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 15:34:27.184395 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184362    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 15:34:27.184601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184579    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 15:34:27.184701 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184681    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 15:34:27.184788 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184593    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 15:34:27.184854 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184785    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 15:34:27.184854 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184800    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 15:34:27.184854 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184581    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4kccb\""
Jul 28 15:34:27.184854 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184792    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 15:34:27.185041 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.184879    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 15:34:27.271146 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-web-config\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271332 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271194    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271332 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271332 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271492 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271492 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56fr\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-kube-api-access-n56fr\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271719 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271610    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271719 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271719 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-out\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.271801 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.271711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.372526 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.372496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.372644 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.372542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-web-config\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.372644 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.372561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.372894 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.372855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.372960 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.372916    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373023 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.372971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373023 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56fr\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-kube-api-access-n56fr\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373122 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373122 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373250 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373250 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373250 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373384 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-out\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373384 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373273    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373983 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373641    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.373983 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.373936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.376003 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.375948    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.376707 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.376665    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-out\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.377221 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.376810    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.377221 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.377111    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.377221 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.377158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-web-config\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.377671 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.377632    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.377864 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.377847    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.378068 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.378048    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.378137 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.378115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.382601 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.382582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56fr\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-kube-api-access-n56fr\") pod \"alertmanager-main-0\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:27.493688 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:27.493644    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:28.195546 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:28.195517    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacb0c31_45fd_47c4_8d9f_b9b86b8bcc62.slice/crio-a639ee39d16719784bc03138a0fa0fa749bebd7b634780db91a1b21c04a2b169 WatchSource:0}: Error finding container a639ee39d16719784bc03138a0fa0fa749bebd7b634780db91a1b21c04a2b169: Status 404 returned error can't find the container with id a639ee39d16719784bc03138a0fa0fa749bebd7b634780db91a1b21c04a2b169
Jul 28 15:34:28.327011 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:28.326983    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75334c30_f4b5_48ef_acb7_d2f449e085f5.slice/crio-124dc1bcee61dc99513b263e4440200f4bafe7873892549ad5ead61592f37e42 WatchSource:0}: Error finding container 124dc1bcee61dc99513b263e4440200f4bafe7873892549ad5ead61592f37e42: Status 404 returned error can't find the container with id 124dc1bcee61dc99513b263e4440200f4bafe7873892549ad5ead61592f37e42
Jul 28 15:34:28.329344 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:28.329324    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:34:28.796470 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:28.796379    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vspq2" event={"ID":"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62","Type":"ContainerStarted","Data":"a639ee39d16719784bc03138a0fa0fa749bebd7b634780db91a1b21c04a2b169"}
Jul 28 15:34:28.798535 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:28.798490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mznsm" event={"ID":"9d0c8519-a071-4afa-a5ab-5bc282690423","Type":"ContainerStarted","Data":"8e9799c7ab54235431e41cc9a59e5aecf74d70eef77307f14d46d7d897f06fd5"}
Jul 28 15:34:28.798662 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:28.798555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mznsm" event={"ID":"9d0c8519-a071-4afa-a5ab-5bc282690423","Type":"ContainerStarted","Data":"a886739a070296bee5765e1213bcec87049eb2ba347b4c0d823e4387bfa8965a"}
Jul 28 15:34:28.798721 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:28.798672    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mznsm"
Jul 28 15:34:28.799842 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:28.799813    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerStarted","Data":"124dc1bcee61dc99513b263e4440200f4bafe7873892549ad5ead61592f37e42"}
Jul 28 15:34:28.801575 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:28.801539    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k99wm" event={"ID":"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d","Type":"ContainerStarted","Data":"ca5d56c86aeb01d966d1f51d6d827462967e7ce8dc4e6f2d9837c059e40d8197"}
Jul 28 15:34:28.818530 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:28.818482    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mznsm" podStartSLOduration=67.314791547 podStartE2EDuration="1m11.818464312s" totalImagesPullingTime="4.503672765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:28.816203057 +0000 UTC m=+105.066260063" watchObservedRunningTime="2026-07-28 15:34:28.818464312 +0000 UTC m=+105.068521312"
Jul 28 15:34:29.069684 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.069543    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-558bc6f974-5kv4r"]
Jul 28 15:34:29.073905 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.073882    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.077897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.077134    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 15:34:29.077897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.077285    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 15:34:29.077897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.077377    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 15:34:29.077897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.077509    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3brd6cl1qqq8o\""
Jul 28 15:34:29.077897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.077548    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 15:34:29.077897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.077722    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 15:34:29.077897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.077841    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mqs6b\""
Jul 28 15:34:29.087709 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.087687    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-558bc6f974-5kv4r"]
Jul 28 15:34:29.190581 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.190491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-grpc-tls\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.190581 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.190554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22xm\" (UniqueName: \"kubernetes.io/projected/fbaa0008-e48f-4b2f-9170-1d2a9af61125-kube-api-access-p22xm\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.190837 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.190594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa0008-e48f-4b2f-9170-1d2a9af61125-metrics-client-ca\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.190837 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.190676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.190837 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.190711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.190837 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.190818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.191036 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.190847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-tls\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.191036 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.190886    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.291603 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.291565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.292084 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.291612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.292084 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.291708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.292084 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.291739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-tls\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.292084 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.291765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.292084 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.291802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-grpc-tls\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.292084 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.291838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22xm\" (UniqueName: \"kubernetes.io/projected/fbaa0008-e48f-4b2f-9170-1d2a9af61125-kube-api-access-p22xm\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.292084 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.291864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa0008-e48f-4b2f-9170-1d2a9af61125-metrics-client-ca\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.294281 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.294204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa0008-e48f-4b2f-9170-1d2a9af61125-metrics-client-ca\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.295180 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.294883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.296523 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.296498    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-tls\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.300975 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.300833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.300975 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.300851    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.301125 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.301073    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-grpc-tls\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.301255 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.301136    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fbaa0008-e48f-4b2f-9170-1d2a9af61125-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.305043 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.303828    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22xm\" (UniqueName: \"kubernetes.io/projected/fbaa0008-e48f-4b2f-9170-1d2a9af61125-kube-api-access-p22xm\") pod \"thanos-querier-558bc6f974-5kv4r\" (UID: \"fbaa0008-e48f-4b2f-9170-1d2a9af61125\") " pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.387211 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.387110    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:29.662600 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.662518    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-558bc6f974-5kv4r"]
Jul 28 15:34:29.665995 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:29.665965    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbaa0008_e48f_4b2f_9170_1d2a9af61125.slice/crio-557f415261c3ce9c6afda896c5a595c89284ca4d87ac155b4cacb5e43f3d37d8 WatchSource:0}: Error finding container 557f415261c3ce9c6afda896c5a595c89284ca4d87ac155b4cacb5e43f3d37d8: Status 404 returned error can't find the container with id 557f415261c3ce9c6afda896c5a595c89284ca4d87ac155b4cacb5e43f3d37d8
Jul 28 15:34:29.808209 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.808150    2575 generic.go:356] "Generic (PLEG): container finished" podID="facb0c31-45fd-47c4-8d9f-b9b86b8bcc62" containerID="f7f9b9b9c701cf6e89d2ad6a3b1f58dc5b13bd45b56fc2786c57586cbb5b91fc" exitCode=0
Jul 28 15:34:29.808397 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.808261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vspq2" event={"ID":"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62","Type":"ContainerDied","Data":"f7f9b9b9c701cf6e89d2ad6a3b1f58dc5b13bd45b56fc2786c57586cbb5b91fc"}
Jul 28 15:34:29.810059 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.810031    2575 generic.go:356] "Generic (PLEG): container finished" podID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerID="ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5" exitCode=0
Jul 28 15:34:29.810195 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.810145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerDied","Data":"ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5"}
Jul 28 15:34:29.811656 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:29.811627    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r" event={"ID":"fbaa0008-e48f-4b2f-9170-1d2a9af61125","Type":"ContainerStarted","Data":"557f415261c3ce9c6afda896c5a595c89284ca4d87ac155b4cacb5e43f3d37d8"}
Jul 28 15:34:30.822107 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:30.822014    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vspq2" event={"ID":"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62","Type":"ContainerStarted","Data":"ef8e75fafaa53ad30bb6ff74715db13801a8bfd73e0ec862ed51e03e87fc3437"}
Jul 28 15:34:30.823305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:30.822550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vspq2" event={"ID":"facb0c31-45fd-47c4-8d9f-b9b86b8bcc62","Type":"ContainerStarted","Data":"96db38cc7c1b1dda6582fba83bbbe8f2c6f1457e418037dd4de26e20093eaccc"}
Jul 28 15:34:30.846432 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:30.846375    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vspq2" podStartSLOduration=3.451555221 podStartE2EDuration="4.846359761s" totalImagesPullingTime="1.315039161s" totalInitContainerRuntime="79.765379ms" isStatefulPod=true podCreationTimestamp="2026-07-28 15:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:30.844257037 +0000 UTC m=+107.094314070" watchObservedRunningTime="2026-07-28 15:34:30.846359761 +0000 UTC m=+107.096416767"
Jul 28 15:34:33.835972 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:33.835943    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerStarted","Data":"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7"}
Jul 28 15:34:33.836410 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:33.836061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerStarted","Data":"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba"}
Jul 28 15:34:33.838355 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:33.838316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k99wm" event={"ID":"f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d","Type":"ContainerStarted","Data":"be6da336a1b8acc900c863af1ced36f5fb933156265b7c07ad570a7d7899fff8"}
Jul 28 15:34:33.840154 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:33.840129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r" event={"ID":"fbaa0008-e48f-4b2f-9170-1d2a9af61125","Type":"ContainerStarted","Data":"d2acdc9099cc2b34983f2d0858e7f27f95f61789ef1edaeb2a591d0dcee39afe"}
Jul 28 15:34:33.840253 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:33.840214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r" event={"ID":"fbaa0008-e48f-4b2f-9170-1d2a9af61125","Type":"ContainerStarted","Data":"09a67bf3a745e04969c4e8877b9e6e49a64a3a5c50b6215ebb7e21e67f5ec816"}
Jul 28 15:34:33.859700 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:33.859646    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k99wm" podStartSLOduration=34.931829296 podStartE2EDuration="50.85962492s" totalImagesPullingTime="15.927795624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:33.858675925 +0000 UTC m=+110.108732929" watchObservedRunningTime="2026-07-28 15:34:33.85962492 +0000 UTC m=+110.109681929"
Jul 28 15:34:34.851513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:34.851484    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r" event={"ID":"fbaa0008-e48f-4b2f-9170-1d2a9af61125","Type":"ContainerStarted","Data":"c7355ad410608e82e9b8dbae171e61bbc100d9dfe8c60e32499bd80971033e91"}
Jul 28 15:34:34.851850 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:34.851584    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r" event={"ID":"fbaa0008-e48f-4b2f-9170-1d2a9af61125","Type":"ContainerStarted","Data":"88103be14a0c6a5ffff36a1c06d292152212199d6e75459994d0e74305c6c4bb"}
Jul 28 15:34:34.851850 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:34.851627    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r" event={"ID":"fbaa0008-e48f-4b2f-9170-1d2a9af61125","Type":"ContainerStarted","Data":"3992940139fc300fd68a57dcdb700674560a8fa20141351eeedd43cb891b62d4"}
Jul 28 15:34:34.855088 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:34.855064    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerStarted","Data":"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6"}
Jul 28 15:34:34.855221 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:34.855136    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerStarted","Data":"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae"}
Jul 28 15:34:34.855221 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:34.855203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerStarted","Data":"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332"}
Jul 28 15:34:34.855346 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:34.855252    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerStarted","Data":"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e"}
Jul 28 15:34:34.887486 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:34.887445    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.154316664 podStartE2EDuration="7.887431282s" totalImagesPullingTime="5.704087114s" totalInitContainerRuntime="29.027504ms" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:34.88553128 +0000 UTC m=+111.135588322" watchObservedRunningTime="2026-07-28 15:34:34.887431282 +0000 UTC m=+111.137488287"
Jul 28 15:34:35.861416 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:35.861381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r" event={"ID":"fbaa0008-e48f-4b2f-9170-1d2a9af61125","Type":"ContainerStarted","Data":"d7d78e79886b4f696c0ff5bcaa4cdac1f7df850e6a6f1799e1d7638e90e4a811"}
Jul 28 15:34:35.861875 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:35.861857    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:35.888778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:35.888732    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r" podStartSLOduration=2.035717239 podStartE2EDuration="6.888719308s" totalImagesPullingTime="4.853002069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:35.886011963 +0000 UTC m=+112.136068970" watchObservedRunningTime="2026-07-28 15:34:35.888719308 +0000 UTC m=+112.138776313"
Jul 28 15:34:36.571341 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:36.571311    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f9ccb895b-vld76"]
Jul 28 15:34:38.813890 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:38.813862    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mznsm"
Jul 28 15:34:41.872876 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:41.872844    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-558bc6f974-5kv4r"
Jul 28 15:34:46.577670 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:46.577643    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:34:53.006364 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:53.006322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:34:53.008772 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:53.008747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab5b87b-81cc-4429-8147-c990db710694-metrics-certs\") pod \"network-metrics-daemon-lx8xf\" (UID: \"9ab5b87b-81cc-4429-8147-c990db710694\") " pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:34:53.265817 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:53.265745    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-25sjx\""
Jul 28 15:34:53.274014 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:53.273988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lx8xf"
Jul 28 15:34:53.455850 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:53.455822    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lx8xf"]
Jul 28 15:34:53.457944 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:34:53.457908    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab5b87b_81cc_4429_8147_c990db710694.slice/crio-ab56e5fb228e89250db69d026010a3c59fd413f5d5909c076b37429dded575fc WatchSource:0}: Error finding container ab56e5fb228e89250db69d026010a3c59fd413f5d5909c076b37429dded575fc: Status 404 returned error can't find the container with id ab56e5fb228e89250db69d026010a3c59fd413f5d5909c076b37429dded575fc
Jul 28 15:34:53.927072 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:53.927027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lx8xf" event={"ID":"9ab5b87b-81cc-4429-8147-c990db710694","Type":"ContainerStarted","Data":"ab56e5fb228e89250db69d026010a3c59fd413f5d5909c076b37429dded575fc"}
Jul 28 15:34:54.932880 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:54.932849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lx8xf" event={"ID":"9ab5b87b-81cc-4429-8147-c990db710694","Type":"ContainerStarted","Data":"113ff5f54c40d44d8262e6d4c8a282ce9c9d154c16854b87114f61fdad19cb4c"}
Jul 28 15:34:55.937886 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:55.937843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lx8xf" event={"ID":"9ab5b87b-81cc-4429-8147-c990db710694","Type":"ContainerStarted","Data":"475433236c2fdfbe245015192b101e3ae775d2fbc29aae487b186f288b4e601c"}
Jul 28 15:34:55.955938 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:34:55.955893    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lx8xf" podStartSLOduration=130.658527156 podStartE2EDuration="2m11.955878697s" totalImagesPullingTime="1.297351541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:55.954701147 +0000 UTC m=+132.204758163" watchObservedRunningTime="2026-07-28 15:34:55.955878697 +0000 UTC m=+132.205935701"
Jul 28 15:35:01.589969 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.589923    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76" podUID="cab2757c-3cb4-4878-924f-776e26a1bb7b" containerName="registry" containerID="cri-o://6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f" gracePeriod=30
Jul 28 15:35:01.832340 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.832316    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:35:01.878860 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.878788    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-kube-api-access-qk77w\") pod \"cab2757c-3cb4-4878-924f-776e26a1bb7b\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") "
Jul 28 15:35:01.878860 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.878827    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-image-registry-private-configuration\") pod \"cab2757c-3cb4-4878-924f-776e26a1bb7b\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") "
Jul 28 15:35:01.878860 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.878851    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cab2757c-3cb4-4878-924f-776e26a1bb7b-ca-trust-extracted\") pod \"cab2757c-3cb4-4878-924f-776e26a1bb7b\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") "
Jul 28 15:35:01.879110 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.878881    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-trusted-ca\") pod \"cab2757c-3cb4-4878-924f-776e26a1bb7b\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") "
Jul 28 15:35:01.879110 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.878910    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-installation-pull-secrets\") pod \"cab2757c-3cb4-4878-924f-776e26a1bb7b\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") "
Jul 28 15:35:01.879110 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.878935    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-bound-sa-token\") pod \"cab2757c-3cb4-4878-924f-776e26a1bb7b\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") "
Jul 28 15:35:01.879110 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.878966    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") pod \"cab2757c-3cb4-4878-924f-776e26a1bb7b\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") "
Jul 28 15:35:01.879431 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.879395    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cab2757c-3cb4-4878-924f-776e26a1bb7b" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:01.881519 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.881473    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cab2757c-3cb4-4878-924f-776e26a1bb7b" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:01.881651 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.881626    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-kube-api-access-qk77w" (OuterVolumeSpecName: "kube-api-access-qk77w") pod "cab2757c-3cb4-4878-924f-776e26a1bb7b" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b"). InnerVolumeSpecName "kube-api-access-qk77w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:35:01.881713 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.881629    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cab2757c-3cb4-4878-924f-776e26a1bb7b" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:01.881768 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.881707    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cab2757c-3cb4-4878-924f-776e26a1bb7b" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:35:01.881768 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.881729    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cab2757c-3cb4-4878-924f-776e26a1bb7b" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:35:01.887569 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.887546    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab2757c-3cb4-4878-924f-776e26a1bb7b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cab2757c-3cb4-4878-924f-776e26a1bb7b" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:35:01.962022 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.961987    2575 generic.go:356] "Generic (PLEG): container finished" podID="cab2757c-3cb4-4878-924f-776e26a1bb7b" containerID="6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f" exitCode=0
Jul 28 15:35:01.962202 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.962033    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76"
Jul 28 15:35:01.962202 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.962039    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76" event={"ID":"cab2757c-3cb4-4878-924f-776e26a1bb7b","Type":"ContainerDied","Data":"6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f"}
Jul 28 15:35:01.962202 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.962080    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9ccb895b-vld76" event={"ID":"cab2757c-3cb4-4878-924f-776e26a1bb7b","Type":"ContainerDied","Data":"bdf655a776c5b86e45249445783612b8575949f2bcd819bdf370e9ac91ced2cb"}
Jul 28 15:35:01.962202 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.962099    2575 scope.go:117] "RemoveContainer" containerID="6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f"
Jul 28 15:35:01.969931 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.969915    2575 scope.go:117] "RemoveContainer" containerID="6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f"
Jul 28 15:35:01.970184 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:35:01.970148    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f\": container with ID starting with 6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f not found: ID does not exist" containerID="6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f"
Jul 28 15:35:01.970230 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.970195    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f"} err="failed to get container status \"6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f\": rpc error: code = NotFound desc = could not find container \"6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f\": container with ID starting with 6d08e0b97c4d5c57b0836750a77733c045f82ef772e60e79ee8e661614aadb2f not found: ID does not exist"
Jul 28 15:35:01.979500 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979483    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-certificates\") pod \"cab2757c-3cb4-4878-924f-776e26a1bb7b\" (UID: \"cab2757c-3cb4-4878-924f-776e26a1bb7b\") "
Jul 28 15:35:01.979652 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979640    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-kube-api-access-qk77w\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:01.979687 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979655    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-image-registry-private-configuration\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:01.979687 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979664    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cab2757c-3cb4-4878-924f-776e26a1bb7b-ca-trust-extracted\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:01.979687 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979672    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-trusted-ca\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:01.979687 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979680    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cab2757c-3cb4-4878-924f-776e26a1bb7b-installation-pull-secrets\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:01.979802 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979688    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-bound-sa-token\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:01.979802 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979698    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:01.979802 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:01.979788    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cab2757c-3cb4-4878-924f-776e26a1bb7b" (UID: "cab2757c-3cb4-4878-924f-776e26a1bb7b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:02.080541 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:02.080513    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cab2757c-3cb4-4878-924f-776e26a1bb7b-registry-certificates\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:02.280996 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:02.280224    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f9ccb895b-vld76"]
Jul 28 15:35:02.283590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:02.283560    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f9ccb895b-vld76"]
Jul 28 15:35:04.259651 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:04.259617    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab2757c-3cb4-4878-924f-776e26a1bb7b" path="/var/lib/kubelet/pods/cab2757c-3cb4-4878-924f-776e26a1bb7b/volumes"
Jul 28 15:35:14.008556 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:14.008524    2575 generic.go:356] "Generic (PLEG): container finished" podID="1f021571-3315-4bf9-889b-110d52285a46" containerID="3a61f0531d6c3150ed60055c049e0ab01fd3d35e04046a2f8772732d3d004039" exitCode=0
Jul 28 15:35:14.008922 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:14.008565    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-d229l" event={"ID":"1f021571-3315-4bf9-889b-110d52285a46","Type":"ContainerDied","Data":"3a61f0531d6c3150ed60055c049e0ab01fd3d35e04046a2f8772732d3d004039"}
Jul 28 15:35:14.008922 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:14.008853    2575 scope.go:117] "RemoveContainer" containerID="3a61f0531d6c3150ed60055c049e0ab01fd3d35e04046a2f8772732d3d004039"
Jul 28 15:35:15.014672 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:15.014636    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-d229l" event={"ID":"1f021571-3315-4bf9-889b-110d52285a46","Type":"ContainerStarted","Data":"7955d345bd9b330a0f9d8bf8208d839d17867297dd5e761ffd257901508be57e"}
Jul 28 15:35:46.416011 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:46.415975    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:35:46.416503 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:46.416288    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="alertmanager" containerID="cri-o://5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba" gracePeriod=120
Jul 28 15:35:46.416503 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:46.416360    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy-metric" containerID="cri-o://1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae" gracePeriod=120
Jul 28 15:35:46.416503 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:46.416385    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy-web" containerID="cri-o://4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e" gracePeriod=120
Jul 28 15:35:46.416503 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:46.416435    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="prom-label-proxy" containerID="cri-o://e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6" gracePeriod=120
Jul 28 15:35:46.416994 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:46.416389    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy" containerID="cri-o://1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332" gracePeriod=120
Jul 28 15:35:46.416994 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:46.416406    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="config-reloader" containerID="cri-o://8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7" gracePeriod=120
Jul 28 15:35:47.138227 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.138194    2575 generic.go:356] "Generic (PLEG): container finished" podID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerID="e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6" exitCode=0
Jul 28 15:35:47.138227 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.138221    2575 generic.go:356] "Generic (PLEG): container finished" podID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerID="1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332" exitCode=0
Jul 28 15:35:47.138227 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.138227    2575 generic.go:356] "Generic (PLEG): container finished" podID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerID="8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7" exitCode=0
Jul 28 15:35:47.138227 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.138234    2575 generic.go:356] "Generic (PLEG): container finished" podID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerID="5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba" exitCode=0
Jul 28 15:35:47.138490 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.138275    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerDied","Data":"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6"}
Jul 28 15:35:47.138490 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.138369    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerDied","Data":"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332"}
Jul 28 15:35:47.138490 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.138397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerDied","Data":"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7"}
Jul 28 15:35:47.138490 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.138423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerDied","Data":"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba"}
Jul 28 15:35:47.668623 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.668598    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:47.846495 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846411    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.846495 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846455    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-metrics-client-ca\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.846495 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846495    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-web-config\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.846778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846519    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-main-tls\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.846778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846537    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-out\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.846778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846556    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-tls-assets\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.846778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846586    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-main-db\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.846778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846607    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-volume\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.847026 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.846993    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:35:47.847075 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.847021    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:47.847129 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.847085    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.847129 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.847120    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-cluster-tls-config\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.847264 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.847158    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-trusted-ca-bundle\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.847264 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.847222    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.847264 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.847254    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n56fr\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-kube-api-access-n56fr\") pod \"75334c30-f4b5-48ef-acb7-d2f449e085f5\" (UID: \"75334c30-f4b5-48ef-acb7-d2f449e085f5\") "
Jul 28 15:35:47.848268 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.847482    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-metrics-client-ca\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.848268 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.847504    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-main-db\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.848268 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.848159    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:47.849482 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.849453    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:35:47.850083 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.850048    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:47.850203 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.850124    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:47.850295 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.850266    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-volume" (OuterVolumeSpecName: "config-volume") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:47.850360 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.850318    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-kube-api-access-n56fr" (OuterVolumeSpecName: "kube-api-access-n56fr") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "kube-api-access-n56fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:35:47.850581 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.850551    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:47.850666 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.850642    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-out" (OuterVolumeSpecName: "config-out") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:35:47.850806 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.850781    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:47.854036 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.853910    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:47.859686 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.859665    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-web-config" (OuterVolumeSpecName: "web-config") pod "75334c30-f4b5-48ef-acb7-d2f449e085f5" (UID: "75334c30-f4b5-48ef-acb7-d2f449e085f5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:47.948271 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948233    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-web-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948271 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948267    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-main-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948271 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948277    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-out\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948286    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-tls-assets\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948294    2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-config-volume\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948303    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948314    2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-cluster-tls-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948322    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75334c30-f4b5-48ef-acb7-d2f449e085f5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948332    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948345    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56fr\" (UniqueName: \"kubernetes.io/projected/75334c30-f4b5-48ef-acb7-d2f449e085f5-kube-api-access-n56fr\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:47.948483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:47.948359    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75334c30-f4b5-48ef-acb7-d2f449e085f5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:35:48.145995 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.145906    2575 generic.go:356] "Generic (PLEG): container finished" podID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerID="1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae" exitCode=0
Jul 28 15:35:48.145995 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.145936    2575 generic.go:356] "Generic (PLEG): container finished" podID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerID="4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e" exitCode=0
Jul 28 15:35:48.146160 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.145987    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerDied","Data":"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae"}
Jul 28 15:35:48.146160 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.146060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerDied","Data":"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e"}
Jul 28 15:35:48.146160 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.145997    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.146160 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.146086    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75334c30-f4b5-48ef-acb7-d2f449e085f5","Type":"ContainerDied","Data":"124dc1bcee61dc99513b263e4440200f4bafe7873892549ad5ead61592f37e42"}
Jul 28 15:35:48.146160 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.146118    2575 scope.go:117] "RemoveContainer" containerID="e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6"
Jul 28 15:35:48.154020 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.153901    2575 scope.go:117] "RemoveContainer" containerID="1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae"
Jul 28 15:35:48.160911 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.160895    2575 scope.go:117] "RemoveContainer" containerID="1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332"
Jul 28 15:35:48.169535 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.169513    2575 scope.go:117] "RemoveContainer" containerID="4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e"
Jul 28 15:35:48.170614 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.170567    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:35:48.176753 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.176735    2575 scope.go:117] "RemoveContainer" containerID="8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7"
Jul 28 15:35:48.181908 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.181889    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:35:48.184046 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.184031    2575 scope.go:117] "RemoveContainer" containerID="5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba"
Jul 28 15:35:48.190484 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.190466    2575 scope.go:117] "RemoveContainer" containerID="ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5"
Jul 28 15:35:48.197011 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.196996    2575 scope.go:117] "RemoveContainer" containerID="e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6"
Jul 28 15:35:48.197279 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:35:48.197259    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6\": container with ID starting with e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6 not found: ID does not exist" containerID="e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6"
Jul 28 15:35:48.197351 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.197292    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6"} err="failed to get container status \"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6\": rpc error: code = NotFound desc = could not find container \"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6\": container with ID starting with e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6 not found: ID does not exist"
Jul 28 15:35:48.197351 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.197317    2575 scope.go:117] "RemoveContainer" containerID="1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae"
Jul 28 15:35:48.197559 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:35:48.197540    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae\": container with ID starting with 1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae not found: ID does not exist" containerID="1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae"
Jul 28 15:35:48.197612 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.197567    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae"} err="failed to get container status \"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae\": rpc error: code = NotFound desc = could not find container \"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae\": container with ID starting with 1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae not found: ID does not exist"
Jul 28 15:35:48.197612 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.197585    2575 scope.go:117] "RemoveContainer" containerID="1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332"
Jul 28 15:35:48.197841 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:35:48.197815    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332\": container with ID starting with 1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332 not found: ID does not exist" containerID="1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332"
Jul 28 15:35:48.197932 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.197846    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332"} err="failed to get container status \"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332\": rpc error: code = NotFound desc = could not find container \"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332\": container with ID starting with 1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332 not found: ID does not exist"
Jul 28 15:35:48.197932 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.197869    2575 scope.go:117] "RemoveContainer" containerID="4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e"
Jul 28 15:35:48.198247 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:35:48.198123    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e\": container with ID starting with 4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e not found: ID does not exist" containerID="4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e"
Jul 28 15:35:48.198247 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.198176    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e"} err="failed to get container status \"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e\": rpc error: code = NotFound desc = could not find container \"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e\": container with ID starting with 4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e not found: ID does not exist"
Jul 28 15:35:48.198247 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.198199    2575 scope.go:117] "RemoveContainer" containerID="8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7"
Jul 28 15:35:48.198467 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:35:48.198451    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7\": container with ID starting with 8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7 not found: ID does not exist" containerID="8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7"
Jul 28 15:35:48.198511 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.198470    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7"} err="failed to get container status \"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7\": rpc error: code = NotFound desc = could not find container \"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7\": container with ID starting with 8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7 not found: ID does not exist"
Jul 28 15:35:48.198511 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.198483    2575 scope.go:117] "RemoveContainer" containerID="5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba"
Jul 28 15:35:48.198705 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:35:48.198690    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba\": container with ID starting with 5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba not found: ID does not exist" containerID="5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba"
Jul 28 15:35:48.198752 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.198708    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba"} err="failed to get container status \"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba\": rpc error: code = NotFound desc = could not find container \"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba\": container with ID starting with 5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba not found: ID does not exist"
Jul 28 15:35:48.198752 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.198720    2575 scope.go:117] "RemoveContainer" containerID="ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5"
Jul 28 15:35:48.198997 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:35:48.198979    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5\": container with ID starting with ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5 not found: ID does not exist" containerID="ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5"
Jul 28 15:35:48.199053 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199005    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5"} err="failed to get container status \"ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5\": rpc error: code = NotFound desc = could not find container \"ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5\": container with ID starting with ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5 not found: ID does not exist"
Jul 28 15:35:48.199053 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199026    2575 scope.go:117] "RemoveContainer" containerID="e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6"
Jul 28 15:35:48.199271 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199252    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6"} err="failed to get container status \"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6\": rpc error: code = NotFound desc = could not find container \"e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6\": container with ID starting with e6f40829839c3ebb46ab5e977e179ceb370732dbf701ca8207105a73a4377bf6 not found: ID does not exist"
Jul 28 15:35:48.199338 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199272    2575 scope.go:117] "RemoveContainer" containerID="1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae"
Jul 28 15:35:48.199473 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199455    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae"} err="failed to get container status \"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae\": rpc error: code = NotFound desc = could not find container \"1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae\": container with ID starting with 1902e3cd8bfd4a2cfe8be2d733d71239220119575d30af125f3db261d444efae not found: ID does not exist"
Jul 28 15:35:48.199538 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199473    2575 scope.go:117] "RemoveContainer" containerID="1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332"
Jul 28 15:35:48.199690 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199670    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332"} err="failed to get container status \"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332\": rpc error: code = NotFound desc = could not find container \"1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332\": container with ID starting with 1bbf7b559662b3cb1f5245cc4235360d87cd46af87acc227f9a34d7c325de332 not found: ID does not exist"
Jul 28 15:35:48.199761 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199693    2575 scope.go:117] "RemoveContainer" containerID="4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e"
Jul 28 15:35:48.199908 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199892    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e"} err="failed to get container status \"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e\": rpc error: code = NotFound desc = could not find container \"4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e\": container with ID starting with 4304d32543a9b4cf3a7541cf1cb8de8c4d492696d380e6c3740790408093136e not found: ID does not exist"
Jul 28 15:35:48.199948 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.199909    2575 scope.go:117] "RemoveContainer" containerID="8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7"
Jul 28 15:35:48.200098 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.200082    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7"} err="failed to get container status \"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7\": rpc error: code = NotFound desc = could not find container \"8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7\": container with ID starting with 8d0bf0c82bf2ba0be296928daa63edadf9290c865c0dae6d58941e24925cedc7 not found: ID does not exist"
Jul 28 15:35:48.200141 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.200098    2575 scope.go:117] "RemoveContainer" containerID="5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba"
Jul 28 15:35:48.200357 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.200341    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba"} err="failed to get container status \"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba\": rpc error: code = NotFound desc = could not find container \"5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba\": container with ID starting with 5e2788ae97f7e8440d985f33bd3cbf5951424008aef20a641af80ccc99d0aeba not found: ID does not exist"
Jul 28 15:35:48.200406 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.200357    2575 scope.go:117] "RemoveContainer" containerID="ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5"
Jul 28 15:35:48.200560 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.200541    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5"} err="failed to get container status \"ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5\": rpc error: code = NotFound desc = could not find container \"ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5\": container with ID starting with ae73d949bb620e9d76bbc53f423d1af32085ec255d2f763b666c232411d47ba5 not found: ID does not exist"
Jul 28 15:35:48.207217 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207196    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:35:48.207487 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207475    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="init-config-reloader"
Jul 28 15:35:48.207534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207490    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="init-config-reloader"
Jul 28 15:35:48.207534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207502    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy"
Jul 28 15:35:48.207534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207508    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy"
Jul 28 15:35:48.207534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207520    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy-web"
Jul 28 15:35:48.207534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207528    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy-web"
Jul 28 15:35:48.207534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207536    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab2757c-3cb4-4878-924f-776e26a1bb7b" containerName="registry"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207541    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab2757c-3cb4-4878-924f-776e26a1bb7b" containerName="registry"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207548    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="prom-label-proxy"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207553    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="prom-label-proxy"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207561    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="config-reloader"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207568    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="config-reloader"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207574    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="alertmanager"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207579    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="alertmanager"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207585    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy-metric"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207591    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy-metric"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207645    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="config-reloader"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207653    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab2757c-3cb4-4878-924f-776e26a1bb7b" containerName="registry"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207659    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="alertmanager"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207665    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy-metric"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207673    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy-web"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207684    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="kube-rbac-proxy"
Jul 28 15:35:48.207702 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.207692    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" containerName="prom-label-proxy"
Jul 28 15:35:48.211328 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.211305    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.215128 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.214731    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 15:35:48.215128 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.214992    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 15:35:48.215128 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.215025    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 15:35:48.215735 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.215493    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 15:35:48.215735 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.215686    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 15:35:48.216940 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.215852    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 15:35:48.216940 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.215901    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4kccb\""
Jul 28 15:35:48.216940 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.216108    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 15:35:48.216940 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.216862    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 15:35:48.219909 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.219760    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 15:35:48.228527 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.228507    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:35:48.255605 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.255581    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75334c30-f4b5-48ef-acb7-d2f449e085f5" path="/var/lib/kubelet/pods/75334c30-f4b5-48ef-acb7-d2f449e085f5/volumes"
Jul 28 15:35:48.350389 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-config-volume\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350389 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-web-config\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn88m\" (UniqueName: \"kubernetes.io/projected/38fb86fa-e411-45da-ba8c-43c74d956a19-kube-api-access-gn88m\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38fb86fa-e411-45da-ba8c-43c74d956a19-tls-assets\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350590 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/38fb86fa-e411-45da-ba8c-43c74d956a19-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350690 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350690 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350789 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350789 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350789 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350764    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38fb86fa-e411-45da-ba8c-43c74d956a19-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350938 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38fb86fa-e411-45da-ba8c-43c74d956a19-config-out\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350938 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.350938 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.350902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38fb86fa-e411-45da-ba8c-43c74d956a19-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.451731 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.451687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.451897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.451739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38fb86fa-e411-45da-ba8c-43c74d956a19-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.451897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.451773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38fb86fa-e411-45da-ba8c-43c74d956a19-config-out\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.451897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.451796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.451897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.451812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38fb86fa-e411-45da-ba8c-43c74d956a19-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.451897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.451829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-config-volume\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.451897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.451845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-web-config\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.451897 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.451882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn88m\" (UniqueName: \"kubernetes.io/projected/38fb86fa-e411-45da-ba8c-43c74d956a19-kube-api-access-gn88m\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.452273 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.452244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38fb86fa-e411-45da-ba8c-43c74d956a19-tls-assets\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.452338 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.452279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/38fb86fa-e411-45da-ba8c-43c74d956a19-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.452393 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.452337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.452393 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.452370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.452493 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.452418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.453521 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.452743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38fb86fa-e411-45da-ba8c-43c74d956a19-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.453521 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.453007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/38fb86fa-e411-45da-ba8c-43c74d956a19-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.453521 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.453156    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38fb86fa-e411-45da-ba8c-43c74d956a19-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.454877 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.454851    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38fb86fa-e411-45da-ba8c-43c74d956a19-config-out\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.455019 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.454990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-web-config\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.455019 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.455003    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.455140 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.455128    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-config-volume\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.455398 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.455376    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.455498 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.455470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.455811 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.455793    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.456203 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.456187    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38fb86fa-e411-45da-ba8c-43c74d956a19-tls-assets\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.456756 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.456736    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38fb86fa-e411-45da-ba8c-43c74d956a19-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.461064 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.461047    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn88m\" (UniqueName: \"kubernetes.io/projected/38fb86fa-e411-45da-ba8c-43c74d956a19-kube-api-access-gn88m\") pod \"alertmanager-main-0\" (UID: \"38fb86fa-e411-45da-ba8c-43c74d956a19\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.522722 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.522691    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:35:48.658262 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:48.658240    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:35:48.660014 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:35:48.659988    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fb86fa_e411_45da_ba8c_43c74d956a19.slice/crio-5526dc296605a805d6668ee462f86bb91dcbe054b458b96f153d2ff92d490465 WatchSource:0}: Error finding container 5526dc296605a805d6668ee462f86bb91dcbe054b458b96f153d2ff92d490465: Status 404 returned error can't find the container with id 5526dc296605a805d6668ee462f86bb91dcbe054b458b96f153d2ff92d490465
Jul 28 15:35:49.151503 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:49.151400    2575 generic.go:356] "Generic (PLEG): container finished" podID="38fb86fa-e411-45da-ba8c-43c74d956a19" containerID="beb7c5e90cb7591af826dbac332a0879dd87c7b44dae3d7d08d4cb68b7ef1483" exitCode=0
Jul 28 15:35:49.151503 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:49.151491    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38fb86fa-e411-45da-ba8c-43c74d956a19","Type":"ContainerDied","Data":"beb7c5e90cb7591af826dbac332a0879dd87c7b44dae3d7d08d4cb68b7ef1483"}
Jul 28 15:35:49.151950 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:49.151595    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38fb86fa-e411-45da-ba8c-43c74d956a19","Type":"ContainerStarted","Data":"5526dc296605a805d6668ee462f86bb91dcbe054b458b96f153d2ff92d490465"}
Jul 28 15:35:50.159457 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.159423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38fb86fa-e411-45da-ba8c-43c74d956a19","Type":"ContainerStarted","Data":"c5b795f1efb70564f805e3d7a6a6a9a62282fc53faab2c263b1a7b2dcf6ca7e6"}
Jul 28 15:35:50.159823 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.159490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38fb86fa-e411-45da-ba8c-43c74d956a19","Type":"ContainerStarted","Data":"454a1fbc04ef2cfb60b5c61716a0565192fbd1f90c029ff2a9dba5a8053db81e"}
Jul 28 15:35:50.159823 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.159515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38fb86fa-e411-45da-ba8c-43c74d956a19","Type":"ContainerStarted","Data":"aab95d6c249e7ac9094369a2a88f90308d63f60ead3bfc169bfd5ebde270b039"}
Jul 28 15:35:50.159823 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.159542    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38fb86fa-e411-45da-ba8c-43c74d956a19","Type":"ContainerStarted","Data":"8dd0f9c7c208fcdaefc63c9dabf2ebd22cdeb029185229cec6977f894681c592"}
Jul 28 15:35:50.159823 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.159579    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38fb86fa-e411-45da-ba8c-43c74d956a19","Type":"ContainerStarted","Data":"ae94a61533d3ded543d91312615ff9aad7b4e90cd121aa94875457f6b76738bf"}
Jul 28 15:35:50.159823 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.159606    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38fb86fa-e411-45da-ba8c-43c74d956a19","Type":"ContainerStarted","Data":"f7f435c41f5b2f6738f51e0723292848428d7b174393e915cfe55945b7b22b8b"}
Jul 28 15:35:50.188583 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.188518    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.172349847 podStartE2EDuration="2.18849712s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.147273ms" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:50.186338213 +0000 UTC m=+186.436395229" watchObservedRunningTime="2026-07-28 15:35:50.18849712 +0000 UTC m=+186.438554127"
Jul 28 15:35:50.406626 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.406586    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7bb8659964-szppw"]
Jul 28 15:35:50.409333 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.409301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.412398 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.412339    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 15:35:50.412398 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.412367    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 15:35:50.412546 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.412462    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-thvwf\""
Jul 28 15:35:50.412653 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.412637    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 15:35:50.412800 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.412784    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 15:35:50.412907 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.412890    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 15:35:50.419497 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.419476    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 15:35:50.425187 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.425151    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bb8659964-szppw"]
Jul 28 15:35:50.568765 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.568730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.568765 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.568772    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-serving-certs-ca-bundle\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.568987 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.568798    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.568987 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.568854    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-federate-client-tls\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.568987 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.568882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-metrics-client-ca\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.568987 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.568929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-telemeter-client-tls\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.568987 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.568960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf974\" (UniqueName: \"kubernetes.io/projected/843c4461-1c35-4d98-80b6-4c80ec996f95-kube-api-access-gf974\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.569147 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.568988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-secret-telemeter-client\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.669725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.669628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-federate-client-tls\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.669725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.669682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-metrics-client-ca\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.669725 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.669721    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-telemeter-client-tls\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.669998 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.669755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf974\" (UniqueName: \"kubernetes.io/projected/843c4461-1c35-4d98-80b6-4c80ec996f95-kube-api-access-gf974\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.669998 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.669780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-secret-telemeter-client\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.669998 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.669813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.669998 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.669849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-serving-certs-ca-bundle\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.669998 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.669876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.670853 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.670820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-metrics-client-ca\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.671587 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.671182    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.671794 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.671739    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/843c4461-1c35-4d98-80b6-4c80ec996f95-serving-certs-ca-bundle\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.673111 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.672964    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.673220 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.673149    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-telemeter-client-tls\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.673312 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.673289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-federate-client-tls\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.673423 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.673360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/843c4461-1c35-4d98-80b6-4c80ec996f95-secret-telemeter-client\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.679860 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.679837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf974\" (UniqueName: \"kubernetes.io/projected/843c4461-1c35-4d98-80b6-4c80ec996f95-kube-api-access-gf974\") pod \"telemeter-client-7bb8659964-szppw\" (UID: \"843c4461-1c35-4d98-80b6-4c80ec996f95\") " pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.721047 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.721009    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bb8659964-szppw"
Jul 28 15:35:50.863873 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:50.863836    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bb8659964-szppw"]
Jul 28 15:35:51.164501 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:51.164460    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bb8659964-szppw" event={"ID":"843c4461-1c35-4d98-80b6-4c80ec996f95","Type":"ContainerStarted","Data":"7090625d2843d151524f0cf0550af850f04354bca5c93c66105a806e88e28805"}
Jul 28 15:35:53.174095 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:53.174054    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bb8659964-szppw" event={"ID":"843c4461-1c35-4d98-80b6-4c80ec996f95","Type":"ContainerStarted","Data":"05790c7d91e1009537cd11d069566d1b326a5cf96f2409d0b1a036061b57c3a5"}
Jul 28 15:35:53.174466 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:53.174130    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bb8659964-szppw" event={"ID":"843c4461-1c35-4d98-80b6-4c80ec996f95","Type":"ContainerStarted","Data":"cf51cf13ce1e082d5d7bc268d5806c03dc504f819d5e27bf48868e57f5974612"}
Jul 28 15:35:53.174466 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:53.174157    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bb8659964-szppw" event={"ID":"843c4461-1c35-4d98-80b6-4c80ec996f95","Type":"ContainerStarted","Data":"d9caf94bd8e8a2a2c7f9d32b881c559b5e33bb4cdadf9bc68702cfe10dbd4397"}
Jul 28 15:35:53.201529 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:35:53.201481    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7bb8659964-szppw" podStartSLOduration=1.449278055 podStartE2EDuration="3.201466011s" totalImagesPullingTime="1.752187956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:53.199705318 +0000 UTC m=+189.449762346" watchObservedRunningTime="2026-07-28 15:35:53.201466011 +0000 UTC m=+189.451523015"
Jul 28 15:37:44.128932 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:37:44.128899    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:37:44.135659 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:37:44.135639    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:37:44.137175 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:37:44.137136    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:37:44.139965 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:37:44.139946    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 15:37:44.144509 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:37:44.144492    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:38:21.288381 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.288342    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"]
Jul 28 15:38:21.291842 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.291820    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.295199 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.295176    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 15:38:21.295313 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.295242    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 15:38:21.295313 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.295288    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dvxq5\""
Jul 28 15:38:21.302669 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.302648    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"]
Jul 28 15:38:21.399003 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.398961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.399225 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.399200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssg5n\" (UniqueName: \"kubernetes.io/projected/419a64b8-db8b-47e4-8f11-825a516005fc-kube-api-access-ssg5n\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.399310 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.399276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.500263 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.500216    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.500434 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.500298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.500434 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.500322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssg5n\" (UniqueName: \"kubernetes.io/projected/419a64b8-db8b-47e4-8f11-825a516005fc-kube-api-access-ssg5n\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.500600 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.500579    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.500647 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.500629    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.510359 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.510330    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssg5n\" (UniqueName: \"kubernetes.io/projected/419a64b8-db8b-47e4-8f11-825a516005fc-kube-api-access-ssg5n\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.600219 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.600185    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:21.725867 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.725837    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"]
Jul 28 15:38:21.728484 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:38:21.728445    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419a64b8_db8b_47e4_8f11_825a516005fc.slice/crio-7792ba9ecd41a4ec1f16f9553eaab34c06cea4301c7f63810c3d78737b5e23c3 WatchSource:0}: Error finding container 7792ba9ecd41a4ec1f16f9553eaab34c06cea4301c7f63810c3d78737b5e23c3: Status 404 returned error can't find the container with id 7792ba9ecd41a4ec1f16f9553eaab34c06cea4301c7f63810c3d78737b5e23c3
Jul 28 15:38:21.730528 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:21.730510    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:38:22.737175 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:22.737134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r" event={"ID":"419a64b8-db8b-47e4-8f11-825a516005fc","Type":"ContainerStarted","Data":"7792ba9ecd41a4ec1f16f9553eaab34c06cea4301c7f63810c3d78737b5e23c3"}
Jul 28 15:38:27.762010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:27.761923    2575 generic.go:356] "Generic (PLEG): container finished" podID="419a64b8-db8b-47e4-8f11-825a516005fc" containerID="9246ce7bd0f1395da5aa1b74867f67d89f320fdb59027932c27a61361abaf808" exitCode=0
Jul 28 15:38:27.762010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:27.761982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r" event={"ID":"419a64b8-db8b-47e4-8f11-825a516005fc","Type":"ContainerDied","Data":"9246ce7bd0f1395da5aa1b74867f67d89f320fdb59027932c27a61361abaf808"}
Jul 28 15:38:30.777082 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:30.777044    2575 generic.go:356] "Generic (PLEG): container finished" podID="419a64b8-db8b-47e4-8f11-825a516005fc" containerID="3e9df68e4759ab20a7247f82f10584c982d071de668cdf60ec19011740acbb8d" exitCode=0
Jul 28 15:38:30.777459 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:30.777112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r" event={"ID":"419a64b8-db8b-47e4-8f11-825a516005fc","Type":"ContainerDied","Data":"3e9df68e4759ab20a7247f82f10584c982d071de668cdf60ec19011740acbb8d"}
Jul 28 15:38:37.809812 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:37.809774    2575 generic.go:356] "Generic (PLEG): container finished" podID="419a64b8-db8b-47e4-8f11-825a516005fc" containerID="67fd08a6af71b8c9cd9cb1fd0f100b7adba9fef7944f7f4e71f19ef669acb159" exitCode=0
Jul 28 15:38:37.810313 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:37.809865    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r" event={"ID":"419a64b8-db8b-47e4-8f11-825a516005fc","Type":"ContainerDied","Data":"67fd08a6af71b8c9cd9cb1fd0f100b7adba9fef7944f7f4e71f19ef669acb159"}
Jul 28 15:38:38.933289 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:38.933262    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:39.060327 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.060284    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssg5n\" (UniqueName: \"kubernetes.io/projected/419a64b8-db8b-47e4-8f11-825a516005fc-kube-api-access-ssg5n\") pod \"419a64b8-db8b-47e4-8f11-825a516005fc\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") "
Jul 28 15:38:39.060519 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.060340    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-bundle\") pod \"419a64b8-db8b-47e4-8f11-825a516005fc\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") "
Jul 28 15:38:39.060519 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.060435    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-util\") pod \"419a64b8-db8b-47e4-8f11-825a516005fc\" (UID: \"419a64b8-db8b-47e4-8f11-825a516005fc\") "
Jul 28 15:38:39.060993 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.060971    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-bundle" (OuterVolumeSpecName: "bundle") pod "419a64b8-db8b-47e4-8f11-825a516005fc" (UID: "419a64b8-db8b-47e4-8f11-825a516005fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:38:39.062473 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.062451    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419a64b8-db8b-47e4-8f11-825a516005fc-kube-api-access-ssg5n" (OuterVolumeSpecName: "kube-api-access-ssg5n") pod "419a64b8-db8b-47e4-8f11-825a516005fc" (UID: "419a64b8-db8b-47e4-8f11-825a516005fc"). InnerVolumeSpecName "kube-api-access-ssg5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:38:39.064623 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.064565    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-util" (OuterVolumeSpecName: "util") pod "419a64b8-db8b-47e4-8f11-825a516005fc" (UID: "419a64b8-db8b-47e4-8f11-825a516005fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:38:39.161543 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.161500    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssg5n\" (UniqueName: \"kubernetes.io/projected/419a64b8-db8b-47e4-8f11-825a516005fc-kube-api-access-ssg5n\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:38:39.161543 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.161534    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-bundle\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:38:39.161543 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.161544    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419a64b8-db8b-47e4-8f11-825a516005fc-util\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:38:39.819519 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.819485    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r"
Jul 28 15:38:39.819689 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.819490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5vqb8r" event={"ID":"419a64b8-db8b-47e4-8f11-825a516005fc","Type":"ContainerDied","Data":"7792ba9ecd41a4ec1f16f9553eaab34c06cea4301c7f63810c3d78737b5e23c3"}
Jul 28 15:38:39.819689 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:39.819609    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7792ba9ecd41a4ec1f16f9553eaab34c06cea4301c7f63810c3d78737b5e23c3"
Jul 28 15:38:44.337155 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.337118    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"]
Jul 28 15:38:44.337538 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.337456    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419a64b8-db8b-47e4-8f11-825a516005fc" containerName="extract"
Jul 28 15:38:44.337538 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.337469    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="419a64b8-db8b-47e4-8f11-825a516005fc" containerName="extract"
Jul 28 15:38:44.337538 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.337483    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419a64b8-db8b-47e4-8f11-825a516005fc" containerName="util"
Jul 28 15:38:44.337538 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.337489    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="419a64b8-db8b-47e4-8f11-825a516005fc" containerName="util"
Jul 28 15:38:44.337538 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.337504    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419a64b8-db8b-47e4-8f11-825a516005fc" containerName="pull"
Jul 28 15:38:44.337538 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.337510    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="419a64b8-db8b-47e4-8f11-825a516005fc" containerName="pull"
Jul 28 15:38:44.337724 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.337565    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="419a64b8-db8b-47e4-8f11-825a516005fc" containerName="extract"
Jul 28 15:38:44.339678 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.339662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:44.342614 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.342586    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 15:38:44.342778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.342631    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 15:38:44.342778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.342679    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-pmkk5\""
Jul 28 15:38:44.342778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.342591    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 15:38:44.353523 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.353500    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"]
Jul 28 15:38:44.401395 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.401364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xkc\" (UniqueName: \"kubernetes.io/projected/9f1de527-7991-43a1-8613-88ec93f5cdc4-kube-api-access-c4xkc\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-q5smh\" (UID: \"9f1de527-7991-43a1-8613-88ec93f5cdc4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:44.401564 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.401472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9f1de527-7991-43a1-8613-88ec93f5cdc4-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-q5smh\" (UID: \"9f1de527-7991-43a1-8613-88ec93f5cdc4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:44.501870 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.501833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9f1de527-7991-43a1-8613-88ec93f5cdc4-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-q5smh\" (UID: \"9f1de527-7991-43a1-8613-88ec93f5cdc4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:44.502018 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.501876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xkc\" (UniqueName: \"kubernetes.io/projected/9f1de527-7991-43a1-8613-88ec93f5cdc4-kube-api-access-c4xkc\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-q5smh\" (UID: \"9f1de527-7991-43a1-8613-88ec93f5cdc4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:44.504240 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.504222    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9f1de527-7991-43a1-8613-88ec93f5cdc4-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-q5smh\" (UID: \"9f1de527-7991-43a1-8613-88ec93f5cdc4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:44.511873 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.511852    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xkc\" (UniqueName: \"kubernetes.io/projected/9f1de527-7991-43a1-8613-88ec93f5cdc4-kube-api-access-c4xkc\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-q5smh\" (UID: \"9f1de527-7991-43a1-8613-88ec93f5cdc4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:44.649914 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.649826    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:44.781840 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.781816    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"]
Jul 28 15:38:44.784690 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:38:44.784665    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1de527_7991_43a1_8613_88ec93f5cdc4.slice/crio-c92e12b6c05f94cd056e0c5bbd228d7dec753fe0f3bb979726d3a1dba9a6c4a3 WatchSource:0}: Error finding container c92e12b6c05f94cd056e0c5bbd228d7dec753fe0f3bb979726d3a1dba9a6c4a3: Status 404 returned error can't find the container with id c92e12b6c05f94cd056e0c5bbd228d7dec753fe0f3bb979726d3a1dba9a6c4a3
Jul 28 15:38:44.840364 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:44.840326    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh" event={"ID":"9f1de527-7991-43a1-8613-88ec93f5cdc4","Type":"ContainerStarted","Data":"c92e12b6c05f94cd056e0c5bbd228d7dec753fe0f3bb979726d3a1dba9a6c4a3"}
Jul 28 15:38:47.856595 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:47.856559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh" event={"ID":"9f1de527-7991-43a1-8613-88ec93f5cdc4","Type":"ContainerStarted","Data":"503fb7d4abfd1296856de7c1505c74cb421fd6b623ed7e21fdb84f60f9ee8467"}
Jul 28 15:38:47.857083 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:47.856687    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:38:47.878564 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:47.878518    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh" podStartSLOduration=1.246901784 podStartE2EDuration="3.878504971s" totalImagesPullingTime="2.631603187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:38:47.876610253 +0000 UTC m=+364.126667257" watchObservedRunningTime="2026-07-28 15:38:47.878504971 +0000 UTC m=+364.128561976"
Jul 28 15:38:48.221188 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.221145    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-2hljt"]
Jul 28 15:38:48.223409 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.223392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.226190 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.226156    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 15:38:48.226323 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.226205    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-kfv9z\""
Jul 28 15:38:48.226540 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.226518    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 15:38:48.234710 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.234685    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-2hljt"]
Jul 28 15:38:48.236140 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.236116    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-certificates\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.236235 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.236197    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-cabundle0\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.236334 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.236232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fttcn\" (UniqueName: \"kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-kube-api-access-fttcn\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.337712 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.337676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-certificates\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.337905 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.337765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-cabundle0\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.337905 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.337805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fttcn\" (UniqueName: \"kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-kube-api-access-fttcn\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.338010 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:38:48.337937    2575 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 15:38:48.338010 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:38:48.337963    2575 secret.go:281] references non-existent secret key: ca.crt
Jul 28 15:38:48.338010 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:38:48.337975    2575 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 15:38:48.338010 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:38:48.337991    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-2hljt: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 15:38:48.338218 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:38:48.338059    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-certificates podName:c5db19e4-92d2-446b-ab70-eeb3c3ed23fe nodeName:}" failed. No retries permitted until 2026-07-28 15:38:48.838036321 +0000 UTC m=+365.088093321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-certificates") pod "keda-operator-77f648d694-2hljt" (UID: "c5db19e4-92d2-446b-ab70-eeb3c3ed23fe") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 15:38:48.338464 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.338446    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-cabundle0\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.346791 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.346768    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fttcn\" (UniqueName: \"kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-kube-api-access-fttcn\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.441571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.441537    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"]
Jul 28 15:38:48.443801 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.443784    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.454830 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.454808    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 15:38:48.482685 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.482597    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"]
Jul 28 15:38:48.539386 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.539333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkjx\" (UniqueName: \"kubernetes.io/projected/347d2a73-9d51-404c-b336-30e487868303-kube-api-access-xkkjx\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.539598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.539462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/347d2a73-9d51-404c-b336-30e487868303-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.539598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.539488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/347d2a73-9d51-404c-b336-30e487868303-certificates\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.640594 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.640558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkjx\" (UniqueName: \"kubernetes.io/projected/347d2a73-9d51-404c-b336-30e487868303-kube-api-access-xkkjx\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.640777 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.640626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/347d2a73-9d51-404c-b336-30e487868303-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.640777 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.640736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/347d2a73-9d51-404c-b336-30e487868303-certificates\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.641006 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.640984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/347d2a73-9d51-404c-b336-30e487868303-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.643322 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.643307    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/347d2a73-9d51-404c-b336-30e487868303-certificates\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.653097 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.653066    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkjx\" (UniqueName: \"kubernetes.io/projected/347d2a73-9d51-404c-b336-30e487868303-kube-api-access-xkkjx\") pod \"keda-metrics-apiserver-55557df8c-n85z9\" (UID: \"347d2a73-9d51-404c-b336-30e487868303\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.660105 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.660084    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vj26j"]
Jul 28 15:38:48.662370 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.662353    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:48.665208 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.665192    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 15:38:48.674950 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.674912    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vj26j"]
Jul 28 15:38:48.741783 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.741682    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9c9\" (UniqueName: \"kubernetes.io/projected/17605eb7-34be-4b36-9844-574e035eedac-kube-api-access-7n9c9\") pod \"keda-admission-56f9798bb4-vj26j\" (UID: \"17605eb7-34be-4b36-9844-574e035eedac\") " pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:48.741965 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.741815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/17605eb7-34be-4b36-9844-574e035eedac-certificates\") pod \"keda-admission-56f9798bb4-vj26j\" (UID: \"17605eb7-34be-4b36-9844-574e035eedac\") " pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:48.753597 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.753555    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:48.842816 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.842774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9c9\" (UniqueName: \"kubernetes.io/projected/17605eb7-34be-4b36-9844-574e035eedac-kube-api-access-7n9c9\") pod \"keda-admission-56f9798bb4-vj26j\" (UID: \"17605eb7-34be-4b36-9844-574e035eedac\") " pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:48.843021 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.842832    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-certificates\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.843021 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.842871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/17605eb7-34be-4b36-9844-574e035eedac-certificates\") pod \"keda-admission-56f9798bb4-vj26j\" (UID: \"17605eb7-34be-4b36-9844-574e035eedac\") " pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:48.845306 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.845278    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c5db19e4-92d2-446b-ab70-eeb3c3ed23fe-certificates\") pod \"keda-operator-77f648d694-2hljt\" (UID: \"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe\") " pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:48.845403 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.845372    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/17605eb7-34be-4b36-9844-574e035eedac-certificates\") pod \"keda-admission-56f9798bb4-vj26j\" (UID: \"17605eb7-34be-4b36-9844-574e035eedac\") " pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:48.853379 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.853352    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9c9\" (UniqueName: \"kubernetes.io/projected/17605eb7-34be-4b36-9844-574e035eedac-kube-api-access-7n9c9\") pod \"keda-admission-56f9798bb4-vj26j\" (UID: \"17605eb7-34be-4b36-9844-574e035eedac\") " pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:48.908580 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.908457    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"]
Jul 28 15:38:48.911721 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:38:48.911282    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347d2a73_9d51_404c_b336_30e487868303.slice/crio-d7cadc0cfbf43756bd782d786fdde6de8adc644484b0223a82151030de82142a WatchSource:0}: Error finding container d7cadc0cfbf43756bd782d786fdde6de8adc644484b0223a82151030de82142a: Status 404 returned error can't find the container with id d7cadc0cfbf43756bd782d786fdde6de8adc644484b0223a82151030de82142a
Jul 28 15:38:48.973293 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:48.973259    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:49.117501 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:49.117469    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vj26j"]
Jul 28 15:38:49.120430 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:38:49.120402    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17605eb7_34be_4b36_9844_574e035eedac.slice/crio-b5a60b2b44b3271309b241c64e4119bcbfb7b3871533efed51a7d750aaf0e6ea WatchSource:0}: Error finding container b5a60b2b44b3271309b241c64e4119bcbfb7b3871533efed51a7d750aaf0e6ea: Status 404 returned error can't find the container with id b5a60b2b44b3271309b241c64e4119bcbfb7b3871533efed51a7d750aaf0e6ea
Jul 28 15:38:49.134994 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:49.134960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:49.274862 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:49.274831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-2hljt"]
Jul 28 15:38:49.277068 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:38:49.277038    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5db19e4_92d2_446b_ab70_eeb3c3ed23fe.slice/crio-da613cf8b28f76e035ddae1803f0d7e325490448dede9a06f7a8d3e89a517fe3 WatchSource:0}: Error finding container da613cf8b28f76e035ddae1803f0d7e325490448dede9a06f7a8d3e89a517fe3: Status 404 returned error can't find the container with id da613cf8b28f76e035ddae1803f0d7e325490448dede9a06f7a8d3e89a517fe3
Jul 28 15:38:49.868927 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:49.868845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9" event={"ID":"347d2a73-9d51-404c-b336-30e487868303","Type":"ContainerStarted","Data":"d7cadc0cfbf43756bd782d786fdde6de8adc644484b0223a82151030de82142a"}
Jul 28 15:38:49.870419 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:49.870354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-vj26j" event={"ID":"17605eb7-34be-4b36-9844-574e035eedac","Type":"ContainerStarted","Data":"b5a60b2b44b3271309b241c64e4119bcbfb7b3871533efed51a7d750aaf0e6ea"}
Jul 28 15:38:49.871754 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:49.871707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-2hljt" event={"ID":"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe","Type":"ContainerStarted","Data":"da613cf8b28f76e035ddae1803f0d7e325490448dede9a06f7a8d3e89a517fe3"}
Jul 28 15:38:51.884518 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:51.884467    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-vj26j" event={"ID":"17605eb7-34be-4b36-9844-574e035eedac","Type":"ContainerStarted","Data":"f9437ed592226565b145a2b365bdd57b5049ff9181cdde5aa31850788fe12dd0"}
Jul 28 15:38:51.884962 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:51.884615    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:38:51.903051 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:51.903001    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-vj26j" podStartSLOduration=2.019063146 podStartE2EDuration="3.902984343s" totalImagesPullingTime="1.883921197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:38:51.901804636 +0000 UTC m=+368.151861640" watchObservedRunningTime="2026-07-28 15:38:51.902984343 +0000 UTC m=+368.153041327"
Jul 28 15:38:53.896492 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:53.896456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-2hljt" event={"ID":"c5db19e4-92d2-446b-ab70-eeb3c3ed23fe","Type":"ContainerStarted","Data":"47413042912b70f30cbb2bf9f2ceb5565b9b655c0e655f6a66706b93ea6c9e66"}
Jul 28 15:38:53.896908 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:53.896616    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:38:53.897895 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:53.897877    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9" event={"ID":"347d2a73-9d51-404c-b336-30e487868303","Type":"ContainerStarted","Data":"e75ac9d503d43cf62a9366f664fa11011b572b2038cfd64c85296dbebeb4a9b5"}
Jul 28 15:38:53.898005 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:53.897994    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:38:53.914649 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:53.914608    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-2hljt" podStartSLOduration=2.204120978 podStartE2EDuration="5.914593486s" totalImagesPullingTime="3.710472508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:38:53.913581573 +0000 UTC m=+370.163638585" watchObservedRunningTime="2026-07-28 15:38:53.914593486 +0000 UTC m=+370.164650490"
Jul 28 15:38:53.930369 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:38:53.930325    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9" podStartSLOduration=1.857887355 podStartE2EDuration="5.930310515s" totalImagesPullingTime="4.07242316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:38:53.929746623 +0000 UTC m=+370.179803629" watchObservedRunningTime="2026-07-28 15:38:53.930310515 +0000 UTC m=+370.180367519"
Jul 28 15:39:04.907527 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:39:04.907494    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n85z9"
Jul 28 15:39:08.864380 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:39:08.864343    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-q5smh"
Jul 28 15:39:12.892685 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:39:12.892652    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-vj26j"
Jul 28 15:39:14.905286 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:39:14.905257    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-2hljt"
Jul 28 15:40:05.226512 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.226476    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-zs2cf"]
Jul 28 15:40:05.229246 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.229223    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:05.232323 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.232297    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 15:40:05.232465 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.232330    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 15:40:05.232465 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.232297    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-l6jnp\""
Jul 28 15:40:05.233752 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.233735    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 15:40:05.241755 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.241729    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zs2cf"]
Jul 28 15:40:05.291100 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.291055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjhp\" (UniqueName: \"kubernetes.io/projected/44837f84-8631-488c-ab11-6b94d1bb0b87-kube-api-access-dcjhp\") pod \"seaweedfs-64568bcd49-zs2cf\" (UID: \"44837f84-8631-488c-ab11-6b94d1bb0b87\") " pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:05.291323 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.291124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44837f84-8631-488c-ab11-6b94d1bb0b87-data\") pod \"seaweedfs-64568bcd49-zs2cf\" (UID: \"44837f84-8631-488c-ab11-6b94d1bb0b87\") " pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:05.391674 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.391631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44837f84-8631-488c-ab11-6b94d1bb0b87-data\") pod \"seaweedfs-64568bcd49-zs2cf\" (UID: \"44837f84-8631-488c-ab11-6b94d1bb0b87\") " pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:05.391844 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.391724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjhp\" (UniqueName: \"kubernetes.io/projected/44837f84-8631-488c-ab11-6b94d1bb0b87-kube-api-access-dcjhp\") pod \"seaweedfs-64568bcd49-zs2cf\" (UID: \"44837f84-8631-488c-ab11-6b94d1bb0b87\") " pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:05.392022 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.392005    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44837f84-8631-488c-ab11-6b94d1bb0b87-data\") pod \"seaweedfs-64568bcd49-zs2cf\" (UID: \"44837f84-8631-488c-ab11-6b94d1bb0b87\") " pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:05.400939 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.400906    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjhp\" (UniqueName: \"kubernetes.io/projected/44837f84-8631-488c-ab11-6b94d1bb0b87-kube-api-access-dcjhp\") pod \"seaweedfs-64568bcd49-zs2cf\" (UID: \"44837f84-8631-488c-ab11-6b94d1bb0b87\") " pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:05.539236 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.539114    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:05.669571 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:05.669461    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zs2cf"]
Jul 28 15:40:05.672117 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:40:05.672090    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44837f84_8631_488c_ab11_6b94d1bb0b87.slice/crio-5b8ec78c86c27ab1c54f10d5484b8d92b50c7b5a8ce668cc510ec13c68d207f5 WatchSource:0}: Error finding container 5b8ec78c86c27ab1c54f10d5484b8d92b50c7b5a8ce668cc510ec13c68d207f5: Status 404 returned error can't find the container with id 5b8ec78c86c27ab1c54f10d5484b8d92b50c7b5a8ce668cc510ec13c68d207f5
Jul 28 15:40:06.221729 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:06.221689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zs2cf" event={"ID":"44837f84-8631-488c-ab11-6b94d1bb0b87","Type":"ContainerStarted","Data":"5b8ec78c86c27ab1c54f10d5484b8d92b50c7b5a8ce668cc510ec13c68d207f5"}
Jul 28 15:40:09.238667 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:09.238632    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zs2cf" event={"ID":"44837f84-8631-488c-ab11-6b94d1bb0b87","Type":"ContainerStarted","Data":"e97aabc8b71c4c89c46e52d14697b4502903774481ba9054799755ca56bb0284"}
Jul 28 15:40:09.239080 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:09.238780    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:40:09.256200 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:09.256142    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-zs2cf" podStartSLOduration=1.627799153 podStartE2EDuration="4.256129299s" totalImagesPullingTime="2.628330146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:09.255660282 +0000 UTC m=+445.505717288" watchObservedRunningTime="2026-07-28 15:40:09.256129299 +0000 UTC m=+445.506186304"
Jul 28 15:40:15.245798 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:40:15.245763    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-zs2cf"
Jul 28 15:41:22.383203 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.383138    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-bghcp"]
Jul 28 15:41:22.386310 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.386288    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:22.389177 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.389139    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 15:41:22.389302 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.389253    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-527nt\""
Jul 28 15:41:22.396194 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.396149    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bghcp"]
Jul 28 15:41:22.452300 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.452269    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0f2487-d84b-434d-bdd6-d610bbb1b218-tls-certs\") pod \"model-serving-api-7b954ffc95-bghcp\" (UID: \"1d0f2487-d84b-434d-bdd6-d610bbb1b218\") " pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:22.452466 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.452327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhw2\" (UniqueName: \"kubernetes.io/projected/1d0f2487-d84b-434d-bdd6-d610bbb1b218-kube-api-access-4bhw2\") pod \"model-serving-api-7b954ffc95-bghcp\" (UID: \"1d0f2487-d84b-434d-bdd6-d610bbb1b218\") " pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:22.553172 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.553140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhw2\" (UniqueName: \"kubernetes.io/projected/1d0f2487-d84b-434d-bdd6-d610bbb1b218-kube-api-access-4bhw2\") pod \"model-serving-api-7b954ffc95-bghcp\" (UID: \"1d0f2487-d84b-434d-bdd6-d610bbb1b218\") " pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:22.553344 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.553249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0f2487-d84b-434d-bdd6-d610bbb1b218-tls-certs\") pod \"model-serving-api-7b954ffc95-bghcp\" (UID: \"1d0f2487-d84b-434d-bdd6-d610bbb1b218\") " pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:22.553386 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:41:22.553347    2575 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 15:41:22.553421 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:41:22.553412    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0f2487-d84b-434d-bdd6-d610bbb1b218-tls-certs podName:1d0f2487-d84b-434d-bdd6-d610bbb1b218 nodeName:}" failed. No retries permitted until 2026-07-28 15:41:23.053391657 +0000 UTC m=+519.303448639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1d0f2487-d84b-434d-bdd6-d610bbb1b218-tls-certs") pod "model-serving-api-7b954ffc95-bghcp" (UID: "1d0f2487-d84b-434d-bdd6-d610bbb1b218") : secret "model-serving-api-tls" not found
Jul 28 15:41:22.564502 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:22.564469    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhw2\" (UniqueName: \"kubernetes.io/projected/1d0f2487-d84b-434d-bdd6-d610bbb1b218-kube-api-access-4bhw2\") pod \"model-serving-api-7b954ffc95-bghcp\" (UID: \"1d0f2487-d84b-434d-bdd6-d610bbb1b218\") " pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:23.057321 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:23.057277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0f2487-d84b-434d-bdd6-d610bbb1b218-tls-certs\") pod \"model-serving-api-7b954ffc95-bghcp\" (UID: \"1d0f2487-d84b-434d-bdd6-d610bbb1b218\") " pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:23.059618 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:23.059595    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0f2487-d84b-434d-bdd6-d610bbb1b218-tls-certs\") pod \"model-serving-api-7b954ffc95-bghcp\" (UID: \"1d0f2487-d84b-434d-bdd6-d610bbb1b218\") " pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:23.297329 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:23.297280    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:23.423214 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:23.423183    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bghcp"]
Jul 28 15:41:23.424498 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:41:23.424471    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0f2487_d84b_434d_bdd6_d610bbb1b218.slice/crio-0dd64313d5f5b4a40c695f0ec30acd1db41870c3ac8cddb5abfebe1f31399c06 WatchSource:0}: Error finding container 0dd64313d5f5b4a40c695f0ec30acd1db41870c3ac8cddb5abfebe1f31399c06: Status 404 returned error can't find the container with id 0dd64313d5f5b4a40c695f0ec30acd1db41870c3ac8cddb5abfebe1f31399c06
Jul 28 15:41:23.558723 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:23.558689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bghcp" event={"ID":"1d0f2487-d84b-434d-bdd6-d610bbb1b218","Type":"ContainerStarted","Data":"0dd64313d5f5b4a40c695f0ec30acd1db41870c3ac8cddb5abfebe1f31399c06"}
Jul 28 15:41:26.574912 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:26.574877    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bghcp" event={"ID":"1d0f2487-d84b-434d-bdd6-d610bbb1b218","Type":"ContainerStarted","Data":"3090a4a93088c64eaec805e10245d0ff2b9eff580ea299a34211101001c8eeb1"}
Jul 28 15:41:26.575331 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:26.574948    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:26.591946 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:26.591894    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-bghcp" podStartSLOduration=2.218483768 podStartE2EDuration="4.591878621s" totalImagesPullingTime="2.373394853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:41:26.591333322 +0000 UTC m=+522.841390328" watchObservedRunningTime="2026-07-28 15:41:26.591878621 +0000 UTC m=+522.841935627"
Jul 28 15:41:37.584011 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:37.583937    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-bghcp"
Jul 28 15:41:38.351013 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.350970    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-dvwl4"]
Jul 28 15:41:38.353480 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.353457    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dvwl4"
Jul 28 15:41:38.362830 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.362805    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dvwl4"]
Jul 28 15:41:38.365105 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.365078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78071dbf-7ee4-41dd-b698-1402f7552a9a-model-cache\") pod \"s3-init-dvwl4\" (UID: \"78071dbf-7ee4-41dd-b698-1402f7552a9a\") " pod="kserve/s3-init-dvwl4"
Jul 28 15:41:38.365228 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.365120    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gq2s\" (UniqueName: \"kubernetes.io/projected/78071dbf-7ee4-41dd-b698-1402f7552a9a-kube-api-access-6gq2s\") pod \"s3-init-dvwl4\" (UID: \"78071dbf-7ee4-41dd-b698-1402f7552a9a\") " pod="kserve/s3-init-dvwl4"
Jul 28 15:41:38.465654 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.465616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78071dbf-7ee4-41dd-b698-1402f7552a9a-model-cache\") pod \"s3-init-dvwl4\" (UID: \"78071dbf-7ee4-41dd-b698-1402f7552a9a\") " pod="kserve/s3-init-dvwl4"
Jul 28 15:41:38.465832 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.465670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gq2s\" (UniqueName: \"kubernetes.io/projected/78071dbf-7ee4-41dd-b698-1402f7552a9a-kube-api-access-6gq2s\") pod \"s3-init-dvwl4\" (UID: \"78071dbf-7ee4-41dd-b698-1402f7552a9a\") " pod="kserve/s3-init-dvwl4"
Jul 28 15:41:38.465974 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.465952    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78071dbf-7ee4-41dd-b698-1402f7552a9a-model-cache\") pod \"s3-init-dvwl4\" (UID: \"78071dbf-7ee4-41dd-b698-1402f7552a9a\") " pod="kserve/s3-init-dvwl4"
Jul 28 15:41:38.474957 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.474925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gq2s\" (UniqueName: \"kubernetes.io/projected/78071dbf-7ee4-41dd-b698-1402f7552a9a-kube-api-access-6gq2s\") pod \"s3-init-dvwl4\" (UID: \"78071dbf-7ee4-41dd-b698-1402f7552a9a\") " pod="kserve/s3-init-dvwl4"
Jul 28 15:41:38.668352 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.668272    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dvwl4"
Jul 28 15:41:38.796345 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:38.796321    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dvwl4"]
Jul 28 15:41:38.798368 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:41:38.798338    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78071dbf_7ee4_41dd_b698_1402f7552a9a.slice/crio-876f83d5385a8dfe9fc66d0adaa9814563ecaaa67589bdcc4651efefe485c9e0 WatchSource:0}: Error finding container 876f83d5385a8dfe9fc66d0adaa9814563ecaaa67589bdcc4651efefe485c9e0: Status 404 returned error can't find the container with id 876f83d5385a8dfe9fc66d0adaa9814563ecaaa67589bdcc4651efefe485c9e0
Jul 28 15:41:39.635992 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:39.635946    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dvwl4" event={"ID":"78071dbf-7ee4-41dd-b698-1402f7552a9a","Type":"ContainerStarted","Data":"876f83d5385a8dfe9fc66d0adaa9814563ecaaa67589bdcc4651efefe485c9e0"}
Jul 28 15:41:42.660927 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:42.660833    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dvwl4" event={"ID":"78071dbf-7ee4-41dd-b698-1402f7552a9a","Type":"ContainerStarted","Data":"3178a313d4ab9a96219166707f99f5699cb4d03bae6b71387f550fd432024b04"}
Jul 28 15:41:44.672650 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:44.672565    2575 generic.go:356] "Generic (PLEG): container finished" podID="78071dbf-7ee4-41dd-b698-1402f7552a9a" containerID="3178a313d4ab9a96219166707f99f5699cb4d03bae6b71387f550fd432024b04" exitCode=0
Jul 28 15:41:44.672650 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:44.672607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dvwl4" event={"ID":"78071dbf-7ee4-41dd-b698-1402f7552a9a","Type":"ContainerDied","Data":"3178a313d4ab9a96219166707f99f5699cb4d03bae6b71387f550fd432024b04"}
Jul 28 15:41:49.701389 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:49.701352    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dvwl4" event={"ID":"78071dbf-7ee4-41dd-b698-1402f7552a9a","Type":"ContainerStarted","Data":"b539be5d00465602760109771884c7e69dfb8156be2078292e2a26860fba76c1"}
Jul 28 15:41:49.721113 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:41:49.721055    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-dvwl4" podStartSLOduration=1.886700582 podStartE2EDuration="11.721038727s" totalImagesPullingTime="7.804134003s" totalInitContainerRuntime="2.030204142s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:41:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:41:49.719314023 +0000 UTC m=+545.969371021" watchObservedRunningTime="2026-07-28 15:41:49.721038727 +0000 UTC m=+545.971095732"
Jul 28 15:42:44.158606 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:44.158578    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:42:44.164499 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:44.164477    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:42:44.164774 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:44.164759    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:42:44.170559 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:44.170542    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:42:54.991430 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:54.991392    2575 generic.go:356] "Generic (PLEG): container finished" podID="78071dbf-7ee4-41dd-b698-1402f7552a9a" containerID="b539be5d00465602760109771884c7e69dfb8156be2078292e2a26860fba76c1" exitCode=0
Jul 28 15:42:54.991806 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:54.991460    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dvwl4" event={"ID":"78071dbf-7ee4-41dd-b698-1402f7552a9a","Type":"ContainerDied","Data":"b539be5d00465602760109771884c7e69dfb8156be2078292e2a26860fba76c1"}
Jul 28 15:42:56.122534 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:56.122510    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dvwl4"
Jul 28 15:42:56.242533 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:56.242502    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78071dbf-7ee4-41dd-b698-1402f7552a9a-model-cache\") pod \"78071dbf-7ee4-41dd-b698-1402f7552a9a\" (UID: \"78071dbf-7ee4-41dd-b698-1402f7552a9a\") "
Jul 28 15:42:56.242706 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:56.242553    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gq2s\" (UniqueName: \"kubernetes.io/projected/78071dbf-7ee4-41dd-b698-1402f7552a9a-kube-api-access-6gq2s\") pod \"78071dbf-7ee4-41dd-b698-1402f7552a9a\" (UID: \"78071dbf-7ee4-41dd-b698-1402f7552a9a\") "
Jul 28 15:42:56.244859 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:56.244830    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78071dbf-7ee4-41dd-b698-1402f7552a9a-kube-api-access-6gq2s" (OuterVolumeSpecName: "kube-api-access-6gq2s") pod "78071dbf-7ee4-41dd-b698-1402f7552a9a" (UID: "78071dbf-7ee4-41dd-b698-1402f7552a9a"). InnerVolumeSpecName "kube-api-access-6gq2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:42:56.262260 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:56.262230    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78071dbf-7ee4-41dd-b698-1402f7552a9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "78071dbf-7ee4-41dd-b698-1402f7552a9a" (UID: "78071dbf-7ee4-41dd-b698-1402f7552a9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:42:56.343747 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:56.343706    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78071dbf-7ee4-41dd-b698-1402f7552a9a-model-cache\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:42:56.343747 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:56.343739    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gq2s\" (UniqueName: \"kubernetes.io/projected/78071dbf-7ee4-41dd-b698-1402f7552a9a-kube-api-access-6gq2s\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:42:57.001522 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:57.001494    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dvwl4"
Jul 28 15:42:57.001667 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:57.001490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dvwl4" event={"ID":"78071dbf-7ee4-41dd-b698-1402f7552a9a","Type":"ContainerDied","Data":"876f83d5385a8dfe9fc66d0adaa9814563ecaaa67589bdcc4651efefe485c9e0"}
Jul 28 15:42:57.001667 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:42:57.001611    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="876f83d5385a8dfe9fc66d0adaa9814563ecaaa67589bdcc4651efefe485c9e0"
Jul 28 15:43:09.173077 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.173000    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"]
Jul 28 15:43:09.173461 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.173396    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78071dbf-7ee4-41dd-b698-1402f7552a9a" containerName="s3-init"
Jul 28 15:43:09.173461 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.173410    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="78071dbf-7ee4-41dd-b698-1402f7552a9a" containerName="s3-init"
Jul 28 15:43:09.173461 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.173421    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78071dbf-7ee4-41dd-b698-1402f7552a9a" containerName="download-hf-model"
Jul 28 15:43:09.173461 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.173427    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="78071dbf-7ee4-41dd-b698-1402f7552a9a" containerName="download-hf-model"
Jul 28 15:43:09.173613 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.173499    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="78071dbf-7ee4-41dd-b698-1402f7552a9a" containerName="s3-init"
Jul 28 15:43:09.175430 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.175406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.178602 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.178580    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8fc0c-predictor-serving-cert\""
Jul 28 15:43:09.178754 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.178612    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:43:09.179941 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.179918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:43:09.180051 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.179918    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s454d\""
Jul 28 15:43:09.180051 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.179966    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8fc0c-kube-rbac-proxy-sar-config\""
Jul 28 15:43:09.186858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.186836    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"]
Jul 28 15:43:09.353310 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.353273    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.353481 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.353317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e84937-6958-481b-8304-422535a23446-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.353481 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.353396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhcql\" (UniqueName: \"kubernetes.io/projected/16e84937-6958-481b-8304-422535a23446-kube-api-access-vhcql\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.454051 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.454014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.454243 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.454057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e84937-6958-481b-8304-422535a23446-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.454243 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.454130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhcql\" (UniqueName: \"kubernetes.io/projected/16e84937-6958-481b-8304-422535a23446-kube-api-access-vhcql\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.454243 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:43:09.454194    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-serving-cert: secret "success-200-isvc-8fc0c-predictor-serving-cert" not found
Jul 28 15:43:09.454385 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:43:09.454271    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls podName:16e84937-6958-481b-8304-422535a23446 nodeName:}" failed. No retries permitted until 2026-07-28 15:43:09.954250389 +0000 UTC m=+626.204307378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls") pod "success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" (UID: "16e84937-6958-481b-8304-422535a23446") : secret "success-200-isvc-8fc0c-predictor-serving-cert" not found
Jul 28 15:43:09.454886 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.454860    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e84937-6958-481b-8304-422535a23446-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.467867 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.467836    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhcql\" (UniqueName: \"kubernetes.io/projected/16e84937-6958-481b-8304-422535a23446-kube-api-access-vhcql\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.470973 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.470948    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"]
Jul 28 15:43:09.481716 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.481690    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.487350 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.487327    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:43:09.487460 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.487422    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 28 15:43:09.488414 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.488382    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:43:09.489300 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.488609    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 28 15:43:09.491664 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.491643    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"]
Jul 28 15:43:09.656529 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.656493    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.656696 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.656556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.656696 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.656584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e64bee5-715e-428a-8a99-33e462d48b18-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.656793 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.656703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.656793 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.656770    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2jx8\" (UniqueName: \"kubernetes.io/projected/8e64bee5-715e-428a-8a99-33e462d48b18-kube-api-access-z2jx8\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.758001 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.757908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.758001 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.757970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2jx8\" (UniqueName: \"kubernetes.io/projected/8e64bee5-715e-428a-8a99-33e462d48b18-kube-api-access-z2jx8\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.758305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.758024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.758305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.758068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.758305 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.758094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e64bee5-715e-428a-8a99-33e462d48b18-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.758498 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.758470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e64bee5-715e-428a-8a99-33e462d48b18-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.758632 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:43:09.758526    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 28 15:43:09.758632 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:43:09.758597    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls podName:8e64bee5-715e-428a-8a99-33e462d48b18 nodeName:}" failed. No retries permitted until 2026-07-28 15:43:10.258577474 +0000 UTC m=+626.508634475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls") pod "isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" (UID: "8e64bee5-715e-428a-8a99-33e462d48b18") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 28 15:43:09.758747 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.758703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.759149 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.759129    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.768288 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.768264    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"]
Jul 28 15:43:09.771101 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.771086    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:09.773054 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.773032    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2jx8\" (UniqueName: \"kubernetes.io/projected/8e64bee5-715e-428a-8a99-33e462d48b18-kube-api-access-z2jx8\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:09.773878 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.773861    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 28 15:43:09.773961 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.773862    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 28 15:43:09.783284 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.783262    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"]
Jul 28 15:43:09.960100 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.960051    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:09.960312 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.960113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:09.960312 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.960141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:09.960312 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.960192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85371-b616-460b-a02b-47e159b61974-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:09.960430 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.960327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85371-b616-460b-a02b-47e159b61974-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:09.960430 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.960369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdv7\" (UniqueName: \"kubernetes.io/projected/abc85371-b616-460b-a02b-47e159b61974-kube-api-access-pwdv7\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:09.962537 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:09.962515    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls\") pod \"success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw\" (UID: \"16e84937-6958-481b-8304-422535a23446\") " pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:10.061430 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.061333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.061430 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.061384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.061430 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.061413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85371-b616-460b-a02b-47e159b61974-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.061705 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.061478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85371-b616-460b-a02b-47e159b61974-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.061705 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.061503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdv7\" (UniqueName: \"kubernetes.io/projected/abc85371-b616-460b-a02b-47e159b61974-kube-api-access-pwdv7\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.061967 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.061943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85371-b616-460b-a02b-47e159b61974-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.062132 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.062113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.062213 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.062157    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.063893 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.063873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85371-b616-460b-a02b-47e159b61974-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.069928 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.069905    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdv7\" (UniqueName: \"kubernetes.io/projected/abc85371-b616-460b-a02b-47e159b61974-kube-api-access-pwdv7\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f7v2d\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.086275 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.086255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:10.090676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.090662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:10.229680 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.229651    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"]
Jul 28 15:43:10.231375 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:43:10.231350    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc85371_b616_460b_a02b_47e159b61974.slice/crio-f4d9b3ee1054dc6781c95b145175665523f37591acb7aad1e048bed74fa20573 WatchSource:0}: Error finding container f4d9b3ee1054dc6781c95b145175665523f37591acb7aad1e048bed74fa20573: Status 404 returned error can't find the container with id f4d9b3ee1054dc6781c95b145175665523f37591acb7aad1e048bed74fa20573
Jul 28 15:43:10.243260 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.243138    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"]
Jul 28 15:43:10.246069 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:43:10.246043    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e84937_6958_481b_8304_422535a23446.slice/crio-5827f7fc735b2243ab333cc3b38db0f1db10f5da9d667cf9f37f97693605a1ee WatchSource:0}: Error finding container 5827f7fc735b2243ab333cc3b38db0f1db10f5da9d667cf9f37f97693605a1ee: Status 404 returned error can't find the container with id 5827f7fc735b2243ab333cc3b38db0f1db10f5da9d667cf9f37f97693605a1ee
Jul 28 15:43:10.263076 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.263052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:10.265365 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.265340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:10.396235 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.396132    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:10.536969 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:10.536943    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"]
Jul 28 15:43:10.539291 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:43:10.539264    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e64bee5_715e_428a_8a99_33e462d48b18.slice/crio-f251964fdaf593e8211ba1e403fef9ac7d3893fe90dfc32a6ade3c22e0a36ac3 WatchSource:0}: Error finding container f251964fdaf593e8211ba1e403fef9ac7d3893fe90dfc32a6ade3c22e0a36ac3: Status 404 returned error can't find the container with id f251964fdaf593e8211ba1e403fef9ac7d3893fe90dfc32a6ade3c22e0a36ac3
Jul 28 15:43:11.070916 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:11.070880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" event={"ID":"16e84937-6958-481b-8304-422535a23446","Type":"ContainerStarted","Data":"5827f7fc735b2243ab333cc3b38db0f1db10f5da9d667cf9f37f97693605a1ee"}
Jul 28 15:43:11.074949 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:11.074913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" event={"ID":"8e64bee5-715e-428a-8a99-33e462d48b18","Type":"ContainerStarted","Data":"1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353"}
Jul 28 15:43:11.075078 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:11.074986    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" event={"ID":"8e64bee5-715e-428a-8a99-33e462d48b18","Type":"ContainerStarted","Data":"f251964fdaf593e8211ba1e403fef9ac7d3893fe90dfc32a6ade3c22e0a36ac3"}
Jul 28 15:43:11.078037 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:11.078010    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" event={"ID":"abc85371-b616-460b-a02b-47e159b61974","Type":"ContainerStarted","Data":"b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3"}
Jul 28 15:43:11.078152 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:11.078070    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" event={"ID":"abc85371-b616-460b-a02b-47e159b61974","Type":"ContainerStarted","Data":"f4d9b3ee1054dc6781c95b145175665523f37591acb7aad1e048bed74fa20573"}
Jul 28 15:43:15.107926 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:15.107837    2575 generic.go:356] "Generic (PLEG): container finished" podID="abc85371-b616-460b-a02b-47e159b61974" containerID="b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3" exitCode=0
Jul 28 15:43:15.108510 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:15.107913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" event={"ID":"abc85371-b616-460b-a02b-47e159b61974","Type":"ContainerDied","Data":"b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3"}
Jul 28 15:43:16.117474 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:16.117390    2575 generic.go:356] "Generic (PLEG): container finished" podID="8e64bee5-715e-428a-8a99-33e462d48b18" containerID="1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353" exitCode=0
Jul 28 15:43:16.117474 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:16.117460    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" event={"ID":"8e64bee5-715e-428a-8a99-33e462d48b18","Type":"ContainerDied","Data":"1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353"}
Jul 28 15:43:36.565326 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:36.565305    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:43:37.246686 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:37.246645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" event={"ID":"8e64bee5-715e-428a-8a99-33e462d48b18","Type":"ContainerStarted","Data":"0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e"}
Jul 28 15:43:37.249371 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:37.249330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" event={"ID":"abc85371-b616-460b-a02b-47e159b61974","Type":"ContainerStarted","Data":"1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25"}
Jul 28 15:43:37.251678 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:37.251616    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" event={"ID":"16e84937-6958-481b-8304-422535a23446","Type":"ContainerStarted","Data":"b97575e8ce9d2f856b4063ffc7eed522911a2ea64f46dc9d756a6f0e0e13e9b6"}
Jul 28 15:43:40.280339 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.280301    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" event={"ID":"16e84937-6958-481b-8304-422535a23446","Type":"ContainerStarted","Data":"aa6954032029f3feaac8209a61840ba05c3ecf136d1cfb86a2cae7064dbc2e5f"}
Jul 28 15:43:40.282748 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.282712    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" event={"ID":"abc85371-b616-460b-a02b-47e159b61974","Type":"ContainerStarted","Data":"6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e"}
Jul 28 15:43:40.285307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.285129    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:40.285307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.285209    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:40.285307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.285245    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:40.285307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.285273    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:40.286491 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.286268    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 15:43:40.286491 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.286418    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 15:43:40.304061 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.304022    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podStartSLOduration=9.781122061 podStartE2EDuration="31.304008295s" totalImagesPullingTime="21.522886234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:43:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:43:40.302632947 +0000 UTC m=+656.552689954" watchObservedRunningTime="2026-07-28 15:43:40.304008295 +0000 UTC m=+656.554065300"
Jul 28 15:43:40.322147 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:40.322102    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podStartSLOduration=2.775501128 podStartE2EDuration="31.322088469s" totalImagesPullingTime="24.741851439s" totalInitContainerRuntime="3.804735902s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:43:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:43:40.321112818 +0000 UTC m=+656.571169826" watchObservedRunningTime="2026-07-28 15:43:40.322088469 +0000 UTC m=+656.572145473"
Jul 28 15:43:41.290574 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:41.290528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" event={"ID":"8e64bee5-715e-428a-8a99-33e462d48b18","Type":"ContainerStarted","Data":"001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128"}
Jul 28 15:43:41.291043 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:41.290991    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:41.291043 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:41.290993    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 15:43:41.291043 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:41.291017    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 15:43:41.312698 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:41.312655    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podStartSLOduration=4.448906984 podStartE2EDuration="32.312640082s" totalImagesPullingTime="23.990183793s" totalInitContainerRuntime="3.873549305s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:43:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:43:41.310081144 +0000 UTC m=+657.560138149" watchObservedRunningTime="2026-07-28 15:43:41.312640082 +0000 UTC m=+657.562697086"
Jul 28 15:43:42.295872 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:42.295830    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 15:43:42.296471 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:42.295947    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 15:43:42.296471 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:42.296205    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:42.297097 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:42.297076    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:43:43.301507 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:43.301464    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:43:47.300833 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:47.300802    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:43:47.301303 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:47.300894    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:43:47.301504 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:47.301473    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 15:43:47.301570 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:47.301546    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 15:43:48.306753 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:48.306726    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:43:48.307238 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:48.307212    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:43:57.302302 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:57.302263    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 15:43:57.302730 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:57.302264    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 15:43:58.307434 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:43:58.307396    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:44:07.302252 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:07.302212    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 15:44:07.302645 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:07.302212    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 15:44:08.307695 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:08.307653    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:44:17.302334 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:17.302295    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 15:44:17.302796 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:17.302293    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 15:44:18.307674 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:18.307637    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:44:27.301658 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:27.301619    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 15:44:27.302313 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:27.302296    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:44:28.308110 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:28.308073    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:44:37.302996 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:37.302911    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:44:38.307828 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:38.307788    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:44:48.307547 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:48.307504    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:44:49.469939 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.469904    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"]
Jul 28 15:44:49.470429 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.470118    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container" containerID="cri-o://b97575e8ce9d2f856b4063ffc7eed522911a2ea64f46dc9d756a6f0e0e13e9b6" gracePeriod=30
Jul 28 15:44:49.470429 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.470216    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kube-rbac-proxy" containerID="cri-o://aa6954032029f3feaac8209a61840ba05c3ecf136d1cfb86a2cae7064dbc2e5f" gracePeriod=30
Jul 28 15:44:49.530039 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.530006    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"]
Jul 28 15:44:49.532802 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.532784    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.535737 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.535712    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4b277-predictor-serving-cert\""
Jul 28 15:44:49.535737 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.535729    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4b277-kube-rbac-proxy-sar-config\""
Jul 28 15:44:49.544474 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.544448    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"]
Jul 28 15:44:49.556476 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.556448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-proxy-tls\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.556628 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.556492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz4kw\" (UniqueName: \"kubernetes.io/projected/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-api-access-bz4kw\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.556628 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.556529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.623679 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.623635    2575 generic.go:356] "Generic (PLEG): container finished" podID="16e84937-6958-481b-8304-422535a23446" containerID="aa6954032029f3feaac8209a61840ba05c3ecf136d1cfb86a2cae7064dbc2e5f" exitCode=2
Jul 28 15:44:49.623825 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.623707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" event={"ID":"16e84937-6958-481b-8304-422535a23446","Type":"ContainerDied","Data":"aa6954032029f3feaac8209a61840ba05c3ecf136d1cfb86a2cae7064dbc2e5f"}
Jul 28 15:44:49.658010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.657964    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-proxy-tls\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.658232 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.658030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz4kw\" (UniqueName: \"kubernetes.io/projected/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-api-access-bz4kw\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.658232 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.658084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.658925 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.658897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.661112 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.661081    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-proxy-tls\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.681574 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.681545    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz4kw\" (UniqueName: \"kubernetes.io/projected/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-api-access-bz4kw\") pod \"success-200-isvc-4b277-predictor-6cf4674d7-4h57h\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") " pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.846472 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.846372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:49.984348 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:49.984315    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"]
Jul 28 15:44:49.985690 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:44:49.985660    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26cc127_e2c3_47ac_b9f8_dfcaf4867550.slice/crio-819915a646e39aafe9911285a816ff96154d7b1bd72185f4e23167de83bbd846 WatchSource:0}: Error finding container 819915a646e39aafe9911285a816ff96154d7b1bd72185f4e23167de83bbd846: Status 404 returned error can't find the container with id 819915a646e39aafe9911285a816ff96154d7b1bd72185f4e23167de83bbd846
Jul 28 15:44:50.631047 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:50.631010    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" event={"ID":"d26cc127-e2c3-47ac-b9f8-dfcaf4867550","Type":"ContainerStarted","Data":"f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4"}
Jul 28 15:44:50.631471 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:50.631079    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" event={"ID":"d26cc127-e2c3-47ac-b9f8-dfcaf4867550","Type":"ContainerStarted","Data":"0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747"}
Jul 28 15:44:50.631471 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:50.631106    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" event={"ID":"d26cc127-e2c3-47ac-b9f8-dfcaf4867550","Type":"ContainerStarted","Data":"819915a646e39aafe9911285a816ff96154d7b1bd72185f4e23167de83bbd846"}
Jul 28 15:44:50.631471 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:50.631137    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:50.653440 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:50.653387    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podStartSLOduration=1.653371696 podStartE2EDuration="1.653371696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:44:50.651395614 +0000 UTC m=+726.901452643" watchObservedRunningTime="2026-07-28 15:44:50.653371696 +0000 UTC m=+726.903428701"
Jul 28 15:44:51.636451 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:51.636417    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:51.637598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:51.637571    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 15:44:52.257096 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.257066    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:44:52.296468 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.296426    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 15:44:52.644098 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.644067    2575 generic.go:356] "Generic (PLEG): container finished" podID="16e84937-6958-481b-8304-422535a23446" containerID="b97575e8ce9d2f856b4063ffc7eed522911a2ea64f46dc9d756a6f0e0e13e9b6" exitCode=0
Jul 28 15:44:52.644532 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.644148    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" event={"ID":"16e84937-6958-481b-8304-422535a23446","Type":"ContainerDied","Data":"b97575e8ce9d2f856b4063ffc7eed522911a2ea64f46dc9d756a6f0e0e13e9b6"}
Jul 28 15:44:52.644888 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.644854    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 15:44:52.721316 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.721292    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:44:52.784154 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.784114    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e84937-6958-481b-8304-422535a23446-kube-rbac-proxy-sar-config\") pod \"16e84937-6958-481b-8304-422535a23446\" (UID: \"16e84937-6958-481b-8304-422535a23446\") "
Jul 28 15:44:52.784154 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.784150    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhcql\" (UniqueName: \"kubernetes.io/projected/16e84937-6958-481b-8304-422535a23446-kube-api-access-vhcql\") pod \"16e84937-6958-481b-8304-422535a23446\" (UID: \"16e84937-6958-481b-8304-422535a23446\") "
Jul 28 15:44:52.784427 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.784246    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls\") pod \"16e84937-6958-481b-8304-422535a23446\" (UID: \"16e84937-6958-481b-8304-422535a23446\") "
Jul 28 15:44:52.784494 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.784472    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e84937-6958-481b-8304-422535a23446-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "16e84937-6958-481b-8304-422535a23446" (UID: "16e84937-6958-481b-8304-422535a23446"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:44:52.786465 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.786436    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16e84937-6958-481b-8304-422535a23446" (UID: "16e84937-6958-481b-8304-422535a23446"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:44:52.786585 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.786483    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e84937-6958-481b-8304-422535a23446-kube-api-access-vhcql" (OuterVolumeSpecName: "kube-api-access-vhcql") pod "16e84937-6958-481b-8304-422535a23446" (UID: "16e84937-6958-481b-8304-422535a23446"). InnerVolumeSpecName "kube-api-access-vhcql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:44:52.885471 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.885376    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e84937-6958-481b-8304-422535a23446-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:44:52.885471 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.885404    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhcql\" (UniqueName: \"kubernetes.io/projected/16e84937-6958-481b-8304-422535a23446-kube-api-access-vhcql\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:44:52.885471 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:52.885416    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e84937-6958-481b-8304-422535a23446-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:44:53.650781 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:53.650742    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw" event={"ID":"16e84937-6958-481b-8304-422535a23446","Type":"ContainerDied","Data":"5827f7fc735b2243ab333cc3b38db0f1db10f5da9d667cf9f37f97693605a1ee"}
Jul 28 15:44:53.650781 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:53.650783    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"
Jul 28 15:44:53.651275 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:53.650843    2575 scope.go:117] "RemoveContainer" containerID="aa6954032029f3feaac8209a61840ba05c3ecf136d1cfb86a2cae7064dbc2e5f"
Jul 28 15:44:53.660071 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:53.660053    2575 scope.go:117] "RemoveContainer" containerID="b97575e8ce9d2f856b4063ffc7eed522911a2ea64f46dc9d756a6f0e0e13e9b6"
Jul 28 15:44:53.673729 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:53.673702    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"]
Jul 28 15:44:53.676071 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:53.676053    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8fc0c-predictor-5697fc6c87-29ghw"]
Jul 28 15:44:54.258437 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:54.258402    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e84937-6958-481b-8304-422535a23446" path="/var/lib/kubelet/pods/16e84937-6958-481b-8304-422535a23446/volumes"
Jul 28 15:44:57.649653 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:57.649627    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:44:57.650086 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:44:57.650061    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 15:45:07.650240 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:07.650199    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 15:45:17.650395 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:17.650347    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 15:45:27.650432 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:27.650388    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 15:45:29.502037 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.501996    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"]
Jul 28 15:45:29.502522 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.502337    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" containerID="cri-o://0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e" gracePeriod=30
Jul 28 15:45:29.502522 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.502399    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kube-rbac-proxy" containerID="cri-o://001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128" gracePeriod=30
Jul 28 15:45:29.520129 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.520105    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"]
Jul 28 15:45:29.520892 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.520876    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container"
Jul 28 15:45:29.520945 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.520893    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container"
Jul 28 15:45:29.520945 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.520901    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kube-rbac-proxy"
Jul 28 15:45:29.520945 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.520907    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kube-rbac-proxy"
Jul 28 15:45:29.521041 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.520978    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kserve-container"
Jul 28 15:45:29.521041 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.520988    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e84937-6958-481b-8304-422535a23446" containerName="kube-rbac-proxy"
Jul 28 15:45:29.523280 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.523265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.526035 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.526009    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-642bb-predictor-serving-cert\""
Jul 28 15:45:29.526144 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.526033    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-642bb-kube-rbac-proxy-sar-config\""
Jul 28 15:45:29.535368 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.535266    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"]
Jul 28 15:45:29.554862 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.554832    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"]
Jul 28 15:45:29.555136 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.555111    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container" containerID="cri-o://1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25" gracePeriod=30
Jul 28 15:45:29.555246 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.555141    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kube-rbac-proxy" containerID="cri-o://6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e" gracePeriod=30
Jul 28 15:45:29.598000 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.597961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4cf\" (UniqueName: \"kubernetes.io/projected/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-api-access-gt4cf\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.598178 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.598051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.598236 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.598176    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.698644 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.698603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4cf\" (UniqueName: \"kubernetes.io/projected/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-api-access-gt4cf\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.698837 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.698651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.698837 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.698729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.698837 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:45:29.698819    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-642bb-predictor-serving-cert: secret "success-200-isvc-642bb-predictor-serving-cert" not found
Jul 28 15:45:29.698976 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:45:29.698873    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls podName:566fe303-800a-4a98-9a8b-f8fc3e2681e3 nodeName:}" failed. No retries permitted until 2026-07-28 15:45:30.19885409 +0000 UTC m=+766.448911078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls") pod "success-200-isvc-642bb-predictor-b88b6c887-hlk59" (UID: "566fe303-800a-4a98-9a8b-f8fc3e2681e3") : secret "success-200-isvc-642bb-predictor-serving-cert" not found
Jul 28 15:45:29.699382 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.699360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.708360 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.708333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4cf\" (UniqueName: \"kubernetes.io/projected/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-api-access-gt4cf\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:29.828263 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.828144    2575 generic.go:356] "Generic (PLEG): container finished" podID="8e64bee5-715e-428a-8a99-33e462d48b18" containerID="001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128" exitCode=2
Jul 28 15:45:29.828263 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.828225    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" event={"ID":"8e64bee5-715e-428a-8a99-33e462d48b18","Type":"ContainerDied","Data":"001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128"}
Jul 28 15:45:29.830441 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.830416    2575 generic.go:356] "Generic (PLEG): container finished" podID="abc85371-b616-460b-a02b-47e159b61974" containerID="6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e" exitCode=2
Jul 28 15:45:29.830562 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:29.830448    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" event={"ID":"abc85371-b616-460b-a02b-47e159b61974","Type":"ContainerDied","Data":"6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e"}
Jul 28 15:45:30.202675 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.202640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:30.205064 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.205046    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls\") pod \"success-200-isvc-642bb-predictor-b88b6c887-hlk59\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") " pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:30.436548 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.436512    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:30.576395 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.576368    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"]
Jul 28 15:45:30.578676 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:45:30.578636    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566fe303_800a_4a98_9a8b_f8fc3e2681e3.slice/crio-68427a8617a8642891ed6f658bfda293dabefcdee84b74c989d94e972a6c3209 WatchSource:0}: Error finding container 68427a8617a8642891ed6f658bfda293dabefcdee84b74c989d94e972a6c3209: Status 404 returned error can't find the container with id 68427a8617a8642891ed6f658bfda293dabefcdee84b74c989d94e972a6c3209
Jul 28 15:45:30.837297 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.837199    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" event={"ID":"566fe303-800a-4a98-9a8b-f8fc3e2681e3","Type":"ContainerStarted","Data":"6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09"}
Jul 28 15:45:30.837297 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.837269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" event={"ID":"566fe303-800a-4a98-9a8b-f8fc3e2681e3","Type":"ContainerStarted","Data":"214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba"}
Jul 28 15:45:30.837297 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.837292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" event={"ID":"566fe303-800a-4a98-9a8b-f8fc3e2681e3","Type":"ContainerStarted","Data":"68427a8617a8642891ed6f658bfda293dabefcdee84b74c989d94e972a6c3209"}
Jul 28 15:45:30.837518 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.837345    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:30.856550 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:30.856496    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podStartSLOduration=1.856475672 podStartE2EDuration="1.856475672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:45:30.855234369 +0000 UTC m=+767.105291376" watchObservedRunningTime="2026-07-28 15:45:30.856475672 +0000 UTC m=+767.106532678"
Jul 28 15:45:31.842487 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:31.842453    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:31.843903 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:31.843864    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 15:45:32.252258 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:32.252211    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 15:45:32.296750 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:32.296709    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 15:45:32.847488 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:32.847445    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 15:45:33.301911 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.301870    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 15:45:33.503451 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.503426    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:45:33.533514 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.533436    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85371-b616-460b-a02b-47e159b61974-proxy-tls\") pod \"abc85371-b616-460b-a02b-47e159b61974\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") "
Jul 28 15:45:33.533514 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.533497    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85371-b616-460b-a02b-47e159b61974-kserve-provision-location\") pod \"abc85371-b616-460b-a02b-47e159b61974\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") "
Jul 28 15:45:33.533677 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.533551    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-kube-rbac-proxy-sar-config\") pod \"abc85371-b616-460b-a02b-47e159b61974\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") "
Jul 28 15:45:33.533677 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.533584    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwdv7\" (UniqueName: \"kubernetes.io/projected/abc85371-b616-460b-a02b-47e159b61974-kube-api-access-pwdv7\") pod \"abc85371-b616-460b-a02b-47e159b61974\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") "
Jul 28 15:45:33.533677 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.533661    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-cabundle-cert\") pod \"abc85371-b616-460b-a02b-47e159b61974\" (UID: \"abc85371-b616-460b-a02b-47e159b61974\") "
Jul 28 15:45:33.533882 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.533824    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc85371-b616-460b-a02b-47e159b61974-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abc85371-b616-460b-a02b-47e159b61974" (UID: "abc85371-b616-460b-a02b-47e159b61974"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:45:33.534055 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.534020    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85371-b616-460b-a02b-47e159b61974-kserve-provision-location\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:33.534055 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.533928    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abc85371-b616-460b-a02b-47e159b61974" (UID: "abc85371-b616-460b-a02b-47e159b61974"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:33.534257 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.534088    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abc85371-b616-460b-a02b-47e159b61974" (UID: "abc85371-b616-460b-a02b-47e159b61974"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:33.535588 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.535557    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc85371-b616-460b-a02b-47e159b61974-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abc85371-b616-460b-a02b-47e159b61974" (UID: "abc85371-b616-460b-a02b-47e159b61974"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:45:33.535671 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.535610    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc85371-b616-460b-a02b-47e159b61974-kube-api-access-pwdv7" (OuterVolumeSpecName: "kube-api-access-pwdv7") pod "abc85371-b616-460b-a02b-47e159b61974" (UID: "abc85371-b616-460b-a02b-47e159b61974"). InnerVolumeSpecName "kube-api-access-pwdv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:45:33.635283 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.635247    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdv7\" (UniqueName: \"kubernetes.io/projected/abc85371-b616-460b-a02b-47e159b61974-kube-api-access-pwdv7\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:33.635283 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.635278    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-cabundle-cert\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:33.635283 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.635288    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85371-b616-460b-a02b-47e159b61974-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:33.635516 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.635298    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85371-b616-460b-a02b-47e159b61974-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:33.853413 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.853328    2575 generic.go:356] "Generic (PLEG): container finished" podID="abc85371-b616-460b-a02b-47e159b61974" containerID="1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25" exitCode=0
Jul 28 15:45:33.853413 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.853366    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" event={"ID":"abc85371-b616-460b-a02b-47e159b61974","Type":"ContainerDied","Data":"1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25"}
Jul 28 15:45:33.853413 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.853387    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"
Jul 28 15:45:33.853922 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.853416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d" event={"ID":"abc85371-b616-460b-a02b-47e159b61974","Type":"ContainerDied","Data":"f4d9b3ee1054dc6781c95b145175665523f37591acb7aad1e048bed74fa20573"}
Jul 28 15:45:33.853922 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.853444    2575 scope.go:117] "RemoveContainer" containerID="6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e"
Jul 28 15:45:33.862076 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.862051    2575 scope.go:117] "RemoveContainer" containerID="1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25"
Jul 28 15:45:33.870266 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.870249    2575 scope.go:117] "RemoveContainer" containerID="b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3"
Jul 28 15:45:33.878801 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.878769    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"]
Jul 28 15:45:33.880367 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.880347    2575 scope.go:117] "RemoveContainer" containerID="6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e"
Jul 28 15:45:33.880699 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:45:33.880674    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e\": container with ID starting with 6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e not found: ID does not exist" containerID="6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e"
Jul 28 15:45:33.880786 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.880710    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e"} err="failed to get container status \"6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e\": rpc error: code = NotFound desc = could not find container \"6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e\": container with ID starting with 6ebded4abd8756509468f4e4e8eb039ad76f070350fdb5ccf55c2af6da8aaa3e not found: ID does not exist"
Jul 28 15:45:33.880786 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.880736    2575 scope.go:117] "RemoveContainer" containerID="1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25"
Jul 28 15:45:33.881040 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:45:33.881013    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25\": container with ID starting with 1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25 not found: ID does not exist" containerID="1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25"
Jul 28 15:45:33.881105 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.881045    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25"} err="failed to get container status \"1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25\": rpc error: code = NotFound desc = could not find container \"1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25\": container with ID starting with 1ba8c026b5ba7eb230654429c6de4135dd72a4c4e02abb578d61a83f032a8a25 not found: ID does not exist"
Jul 28 15:45:33.881105 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.881067    2575 scope.go:117] "RemoveContainer" containerID="b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3"
Jul 28 15:45:33.881484 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:45:33.881461    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3\": container with ID starting with b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3 not found: ID does not exist" containerID="b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3"
Jul 28 15:45:33.885190 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.881606    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3"} err="failed to get container status \"b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3\": rpc error: code = NotFound desc = could not find container \"b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3\": container with ID starting with b6a34c52da263909d148d6c574c594d25c07714a9d907c0cd9e91978dcaf61f3 not found: ID does not exist"
Jul 28 15:45:33.885190 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:33.885121    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f7v2d"]
Jul 28 15:45:34.257967 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.257930    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc85371-b616-460b-a02b-47e159b61974" path="/var/lib/kubelet/pods/abc85371-b616-460b-a02b-47e159b61974/volumes"
Jul 28 15:45:34.856135 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.856108    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:45:34.859335 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.859309    2575 generic.go:356] "Generic (PLEG): container finished" podID="8e64bee5-715e-428a-8a99-33e462d48b18" containerID="0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e" exitCode=0
Jul 28 15:45:34.859435 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.859364    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"
Jul 28 15:45:34.859435 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.859384    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" event={"ID":"8e64bee5-715e-428a-8a99-33e462d48b18","Type":"ContainerDied","Data":"0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e"}
Jul 28 15:45:34.859521 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.859455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp" event={"ID":"8e64bee5-715e-428a-8a99-33e462d48b18","Type":"ContainerDied","Data":"f251964fdaf593e8211ba1e403fef9ac7d3893fe90dfc32a6ade3c22e0a36ac3"}
Jul 28 15:45:34.859521 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.859495    2575 scope.go:117] "RemoveContainer" containerID="001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128"
Jul 28 15:45:34.867388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.867368    2575 scope.go:117] "RemoveContainer" containerID="0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e"
Jul 28 15:45:34.875149 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.875130    2575 scope.go:117] "RemoveContainer" containerID="1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353"
Jul 28 15:45:34.884656 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.884636    2575 scope.go:117] "RemoveContainer" containerID="001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128"
Jul 28 15:45:34.885000 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:45:34.884981    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128\": container with ID starting with 001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128 not found: ID does not exist" containerID="001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128"
Jul 28 15:45:34.885078 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.885008    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128"} err="failed to get container status \"001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128\": rpc error: code = NotFound desc = could not find container \"001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128\": container with ID starting with 001073c7ae99bc855dc09b07fa40149982484792b4f5f22d8e889955ebc24128 not found: ID does not exist"
Jul 28 15:45:34.885078 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.885024    2575 scope.go:117] "RemoveContainer" containerID="0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e"
Jul 28 15:45:34.885312 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:45:34.885292    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e\": container with ID starting with 0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e not found: ID does not exist" containerID="0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e"
Jul 28 15:45:34.885403 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.885317    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e"} err="failed to get container status \"0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e\": rpc error: code = NotFound desc = could not find container \"0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e\": container with ID starting with 0a69f438de81f24e5416dc32e3f53906a4a785d639a002c8f1a3e43da323130e not found: ID does not exist"
Jul 28 15:45:34.885403 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.885333    2575 scope.go:117] "RemoveContainer" containerID="1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353"
Jul 28 15:45:34.885600 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:45:34.885580    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353\": container with ID starting with 1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353 not found: ID does not exist" containerID="1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353"
Jul 28 15:45:34.885653 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.885605    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353"} err="failed to get container status \"1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353\": rpc error: code = NotFound desc = could not find container \"1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353\": container with ID starting with 1930a5201e18aaff54598147e7eed2fdc29bb2ae42f870ac79ceef59d6153353 not found: ID does not exist"
Jul 28 15:45:34.944986 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.944957    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-kube-rbac-proxy-sar-config\") pod \"8e64bee5-715e-428a-8a99-33e462d48b18\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") "
Jul 28 15:45:34.945157 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.944997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-cabundle-cert\") pod \"8e64bee5-715e-428a-8a99-33e462d48b18\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") "
Jul 28 15:45:34.945157 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.945042    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e64bee5-715e-428a-8a99-33e462d48b18-kserve-provision-location\") pod \"8e64bee5-715e-428a-8a99-33e462d48b18\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") "
Jul 28 15:45:34.945157 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.945088    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2jx8\" (UniqueName: \"kubernetes.io/projected/8e64bee5-715e-428a-8a99-33e462d48b18-kube-api-access-z2jx8\") pod \"8e64bee5-715e-428a-8a99-33e462d48b18\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") "
Jul 28 15:45:34.945157 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.945112    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls\") pod \"8e64bee5-715e-428a-8a99-33e462d48b18\" (UID: \"8e64bee5-715e-428a-8a99-33e462d48b18\") "
Jul 28 15:45:34.945429 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.945400    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e64bee5-715e-428a-8a99-33e462d48b18" (UID: "8e64bee5-715e-428a-8a99-33e462d48b18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:34.945473 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.945435    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e64bee5-715e-428a-8a99-33e462d48b18-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e64bee5-715e-428a-8a99-33e462d48b18" (UID: "8e64bee5-715e-428a-8a99-33e462d48b18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:45:34.945473 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.945455    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e64bee5-715e-428a-8a99-33e462d48b18" (UID: "8e64bee5-715e-428a-8a99-33e462d48b18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:34.947258 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.947231    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e64bee5-715e-428a-8a99-33e462d48b18" (UID: "8e64bee5-715e-428a-8a99-33e462d48b18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:45:34.947342 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:34.947289    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e64bee5-715e-428a-8a99-33e462d48b18-kube-api-access-z2jx8" (OuterVolumeSpecName: "kube-api-access-z2jx8") pod "8e64bee5-715e-428a-8a99-33e462d48b18" (UID: "8e64bee5-715e-428a-8a99-33e462d48b18"). InnerVolumeSpecName "kube-api-access-z2jx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:45:35.046712 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:35.046677    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.046712 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:35.046707    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e64bee5-715e-428a-8a99-33e462d48b18-cabundle-cert\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.046712 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:35.046717    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e64bee5-715e-428a-8a99-33e462d48b18-kserve-provision-location\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.046953 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:35.046726    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2jx8\" (UniqueName: \"kubernetes.io/projected/8e64bee5-715e-428a-8a99-33e462d48b18-kube-api-access-z2jx8\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.046953 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:35.046737    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e64bee5-715e-428a-8a99-33e462d48b18-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:45:35.184947 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:35.184908    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"]
Jul 28 15:45:35.188222 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:35.188179    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-6knsp"]
Jul 28 15:45:36.257221 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:36.257185    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" path="/var/lib/kubelet/pods/8e64bee5-715e-428a-8a99-33e462d48b18/volumes"
Jul 28 15:45:37.651083 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:37.651051    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:45:37.852458 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:37.852427    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:45:37.853061 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:37.853036    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 15:45:47.853767 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:47.853724    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 15:45:57.853705 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:45:57.853661    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 15:46:07.853518 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:46:07.853435    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 15:46:17.853922 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:46:17.853892    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:47:44.189885 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:47:44.189851    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:47:44.194262 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:47:44.194244    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:47:44.196124 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:47:44.196105    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:47:44.200277 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:47:44.200259    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:52:44.222225 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:52:44.222200    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:52:44.223462 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:52:44.223431    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:52:44.228197 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:52:44.228160    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:52:44.229493 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:52:44.229474    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:54:14.431987 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.431952    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"]
Jul 28 15:54:14.432488 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.432268    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container" containerID="cri-o://0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747" gracePeriod=30
Jul 28 15:54:14.432488 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.432271    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kube-rbac-proxy" containerID="cri-o://f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4" gracePeriod=30
Jul 28 15:54:14.509856 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.509827    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"]
Jul 28 15:54:14.510299 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510282    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="storage-initializer"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510300    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="storage-initializer"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510318    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kube-rbac-proxy"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510326    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kube-rbac-proxy"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510337    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kube-rbac-proxy"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510345    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kube-rbac-proxy"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510361    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="storage-initializer"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510366    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="storage-initializer"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510380    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510389    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container"
Jul 28 15:54:14.510394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510397    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container"
Jul 28 15:54:14.510783 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510403    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container"
Jul 28 15:54:14.510783 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510468    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kube-rbac-proxy"
Jul 28 15:54:14.510783 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510479    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kserve-container"
Jul 28 15:54:14.510783 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510489    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e64bee5-715e-428a-8a99-33e462d48b18" containerName="kserve-container"
Jul 28 15:54:14.510783 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.510500    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc85371-b616-460b-a02b-47e159b61974" containerName="kube-rbac-proxy"
Jul 28 15:54:14.513867 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.513849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.517009 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.516987    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ad35d-kube-rbac-proxy-sar-config\""
Jul 28 15:54:14.517330 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.517263    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ad35d-predictor-serving-cert\""
Jul 28 15:54:14.526352 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.526330    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"]
Jul 28 15:54:14.630624 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.630592    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fscr2\" (UniqueName: \"kubernetes.io/projected/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-api-access-fscr2\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.630778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.630673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-proxy-tls\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.630778 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.630722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.732215 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.732183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fscr2\" (UniqueName: \"kubernetes.io/projected/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-api-access-fscr2\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.732388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.732247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-proxy-tls\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.732388 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.732294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.732997 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.732976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.734845 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.734820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-proxy-tls\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.746614 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.746584    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fscr2\" (UniqueName: \"kubernetes.io/projected/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-api-access-fscr2\") pod \"success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") " pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.827398 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.827368    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:14.962855 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:14.962830    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"]
Jul 28 15:54:14.963935 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:54:14.963909    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ef5e1a_bb17_48f6_92ff_05e6a48c1163.slice/crio-7f3f39d4fee8b85dc5c3ddba1bc40a4f57213af32e008b4895003f59b95300d6 WatchSource:0}: Error finding container 7f3f39d4fee8b85dc5c3ddba1bc40a4f57213af32e008b4895003f59b95300d6: Status 404 returned error can't find the container with id 7f3f39d4fee8b85dc5c3ddba1bc40a4f57213af32e008b4895003f59b95300d6
Jul 28 15:54:15.270621 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:15.270545    2575 generic.go:356] "Generic (PLEG): container finished" podID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerID="f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4" exitCode=2
Jul 28 15:54:15.270786 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:15.270620    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" event={"ID":"d26cc127-e2c3-47ac-b9f8-dfcaf4867550","Type":"ContainerDied","Data":"f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4"}
Jul 28 15:54:15.272244 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:15.272224    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" event={"ID":"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163","Type":"ContainerStarted","Data":"d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792"}
Jul 28 15:54:15.272339 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:15.272272    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" event={"ID":"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163","Type":"ContainerStarted","Data":"c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39"}
Jul 28 15:54:15.272339 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:15.272297    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" event={"ID":"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163","Type":"ContainerStarted","Data":"7f3f39d4fee8b85dc5c3ddba1bc40a4f57213af32e008b4895003f59b95300d6"}
Jul 28 15:54:15.272339 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:15.272330    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:15.294696 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:15.294648    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podStartSLOduration=1.294636117 podStartE2EDuration="1.294636117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:54:15.29304445 +0000 UTC m=+1291.543101448" watchObservedRunningTime="2026-07-28 15:54:15.294636117 +0000 UTC m=+1291.544693122"
Jul 28 15:54:16.276964 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:16.276928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:16.278473 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:16.278439    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 15:54:17.284136 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.284091    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 15:54:17.670910 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.670884    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:54:17.756998 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.756971    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-proxy-tls\") pod \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") "
Jul 28 15:54:17.757141 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.757042    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz4kw\" (UniqueName: \"kubernetes.io/projected/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-api-access-bz4kw\") pod \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") "
Jul 28 15:54:17.757141 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.757091    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-rbac-proxy-sar-config\") pod \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\" (UID: \"d26cc127-e2c3-47ac-b9f8-dfcaf4867550\") "
Jul 28 15:54:17.757470 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.757440    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d26cc127-e2c3-47ac-b9f8-dfcaf4867550" (UID: "d26cc127-e2c3-47ac-b9f8-dfcaf4867550"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:54:17.759104 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.759078    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d26cc127-e2c3-47ac-b9f8-dfcaf4867550" (UID: "d26cc127-e2c3-47ac-b9f8-dfcaf4867550"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:54:17.759203 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.759117    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-api-access-bz4kw" (OuterVolumeSpecName: "kube-api-access-bz4kw") pod "d26cc127-e2c3-47ac-b9f8-dfcaf4867550" (UID: "d26cc127-e2c3-47ac-b9f8-dfcaf4867550"). InnerVolumeSpecName "kube-api-access-bz4kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:54:17.858055 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.857984    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:54:17.858055 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.858006    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz4kw\" (UniqueName: \"kubernetes.io/projected/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-api-access-bz4kw\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:54:17.858055 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:17.858016    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26cc127-e2c3-47ac-b9f8-dfcaf4867550-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:54:18.290267 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.290234    2575 generic.go:356] "Generic (PLEG): container finished" podID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerID="0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747" exitCode=0
Jul 28 15:54:18.290602 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.290281    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"
Jul 28 15:54:18.290602 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.290281    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" event={"ID":"d26cc127-e2c3-47ac-b9f8-dfcaf4867550","Type":"ContainerDied","Data":"0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747"}
Jul 28 15:54:18.290602 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.290347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" event={"ID":"d26cc127-e2c3-47ac-b9f8-dfcaf4867550","Type":"ContainerDied","Data":"819915a646e39aafe9911285a816ff96154d7b1bd72185f4e23167de83bbd846"}
Jul 28 15:54:18.290602 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.290386    2575 scope.go:117] "RemoveContainer" containerID="f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4"
Jul 28 15:54:18.298599 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.298579    2575 scope.go:117] "RemoveContainer" containerID="0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747"
Jul 28 15:54:18.305789 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.305769    2575 scope.go:117] "RemoveContainer" containerID="f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4"
Jul 28 15:54:18.306016 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:54:18.305994    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4\": container with ID starting with f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4 not found: ID does not exist" containerID="f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4"
Jul 28 15:54:18.306093 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.306027    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4"} err="failed to get container status \"f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4\": rpc error: code = NotFound desc = could not find container \"f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4\": container with ID starting with f6dadd741fbe1139ba6d7b6b73d1a36e5a48c941e23d9f0a455a053c5417eed4 not found: ID does not exist"
Jul 28 15:54:18.306093 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.306049    2575 scope.go:117] "RemoveContainer" containerID="0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747"
Jul 28 15:54:18.306319 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:54:18.306299    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747\": container with ID starting with 0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747 not found: ID does not exist" containerID="0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747"
Jul 28 15:54:18.306356 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.306324    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747"} err="failed to get container status \"0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747\": rpc error: code = NotFound desc = could not find container \"0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747\": container with ID starting with 0396eb1fce69e7d3aef5482967f40eeb3c0ea0f12494d7c61e60d55e7c3ed747 not found: ID does not exist"
Jul 28 15:54:18.311886 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.311866    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"]
Jul 28 15:54:18.317802 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.317779    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h"]
Jul 28 15:54:18.646307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.646217    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 15:54:18.651376 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:18.651351    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b277-predictor-6cf4674d7-4h57h" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: i/o timeout"
Jul 28 15:54:20.257230 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:20.257197    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" path="/var/lib/kubelet/pods/d26cc127-e2c3-47ac-b9f8-dfcaf4867550/volumes"
Jul 28 15:54:22.288234 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:22.288207    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:54:22.288677 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:22.288614    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 15:54:32.288665 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:32.288604    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 15:54:42.288598 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:42.288561    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 15:54:44.606753 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.606668    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"]
Jul 28 15:54:44.607115 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.607047    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kube-rbac-proxy"
Jul 28 15:54:44.607115 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.607057    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kube-rbac-proxy"
Jul 28 15:54:44.607115 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.607078    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container"
Jul 28 15:54:44.607115 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.607084    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container"
Jul 28 15:54:44.607272 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.607133    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kserve-container"
Jul 28 15:54:44.607272 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.607143    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26cc127-e2c3-47ac-b9f8-dfcaf4867550" containerName="kube-rbac-proxy"
Jul 28 15:54:44.611275 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.611254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:44.615353 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.615330    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-12202-kube-rbac-proxy-sar-config\""
Jul 28 15:54:44.615627 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.615606    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-12202-predictor-serving-cert\""
Jul 28 15:54:44.629050 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.629025    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"]
Jul 28 15:54:44.629332 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.629286    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container" containerID="cri-o://214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba" gracePeriod=30
Jul 28 15:54:44.629421 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.629318    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kube-rbac-proxy" containerID="cri-o://6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09" gracePeriod=30
Jul 28 15:54:44.631511 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.631490    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"]
Jul 28 15:54:44.682858 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.682834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:44.682991 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.682875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:44.682991 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.682916    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkns\" (UniqueName: \"kubernetes.io/projected/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-api-access-zdkns\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:44.784058 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.784029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:44.784245 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.784072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:44.784315 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.784295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkns\" (UniqueName: \"kubernetes.io/projected/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-api-access-zdkns\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:44.784436 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:54:44.784410    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-12202-predictor-serving-cert: secret "success-200-isvc-12202-predictor-serving-cert" not found
Jul 28 15:54:44.784518 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:54:44.784506    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls podName:7c7c7959-59bd-492e-87f4-1099cf7b46b8 nodeName:}" failed. No retries permitted until 2026-07-28 15:54:45.284481584 +0000 UTC m=+1321.534538587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls") pod "success-200-isvc-12202-predictor-f74599657-2zkrq" (UID: "7c7c7959-59bd-492e-87f4-1099cf7b46b8") : secret "success-200-isvc-12202-predictor-serving-cert" not found
Jul 28 15:54:44.784783 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.784760    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:44.810348 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:44.810322    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkns\" (UniqueName: \"kubernetes.io/projected/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-api-access-zdkns\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:45.289797 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:45.289761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:45.292204 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:45.292184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls\") pod \"success-200-isvc-12202-predictor-f74599657-2zkrq\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") " pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:45.415874 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:45.415844    2575 generic.go:356] "Generic (PLEG): container finished" podID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerID="6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09" exitCode=2
Jul 28 15:54:45.416032 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:45.415928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" event={"ID":"566fe303-800a-4a98-9a8b-f8fc3e2681e3","Type":"ContainerDied","Data":"6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09"}
Jul 28 15:54:45.521223 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:45.521189    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:45.660341 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:45.660320    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"]
Jul 28 15:54:45.662436 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:54:45.662395    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7c7959_59bd_492e_87f4_1099cf7b46b8.slice/crio-05b4781bfc5aef9cabf1cce060a3f6130b2d1b786af51e05f1bf23a5251e6f34 WatchSource:0}: Error finding container 05b4781bfc5aef9cabf1cce060a3f6130b2d1b786af51e05f1bf23a5251e6f34: Status 404 returned error can't find the container with id 05b4781bfc5aef9cabf1cce060a3f6130b2d1b786af51e05f1bf23a5251e6f34
Jul 28 15:54:46.422857 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:46.422824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" event={"ID":"7c7c7959-59bd-492e-87f4-1099cf7b46b8","Type":"ContainerStarted","Data":"30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e"}
Jul 28 15:54:46.423039 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:46.422885    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:46.423039 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:46.422913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" event={"ID":"7c7c7959-59bd-492e-87f4-1099cf7b46b8","Type":"ContainerStarted","Data":"5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827"}
Jul 28 15:54:46.423039 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:46.422934    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" event={"ID":"7c7c7959-59bd-492e-87f4-1099cf7b46b8","Type":"ContainerStarted","Data":"05b4781bfc5aef9cabf1cce060a3f6130b2d1b786af51e05f1bf23a5251e6f34"}
Jul 28 15:54:46.423039 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:46.422955    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:46.424394 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:46.424370    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 15:54:46.450582 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:46.450539    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podStartSLOduration=2.450527519 podStartE2EDuration="2.450527519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:54:46.448051879 +0000 UTC m=+1322.698108900" watchObservedRunningTime="2026-07-28 15:54:46.450527519 +0000 UTC m=+1322.700584585"
Jul 28 15:54:47.427293 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:47.427253    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 15:54:47.847841 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:47.847797    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 28 15:54:47.853148 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:47.853124    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 15:54:48.173177 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.173142    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:54:48.215102 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.215067    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls\") pod \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") "
Jul 28 15:54:48.215271 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.215125    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-rbac-proxy-sar-config\") pod \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") "
Jul 28 15:54:48.215271 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.215184    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt4cf\" (UniqueName: \"kubernetes.io/projected/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-api-access-gt4cf\") pod \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\" (UID: \"566fe303-800a-4a98-9a8b-f8fc3e2681e3\") "
Jul 28 15:54:48.215535 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.215512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "566fe303-800a-4a98-9a8b-f8fc3e2681e3" (UID: "566fe303-800a-4a98-9a8b-f8fc3e2681e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:54:48.217086 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.217060    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "566fe303-800a-4a98-9a8b-f8fc3e2681e3" (UID: "566fe303-800a-4a98-9a8b-f8fc3e2681e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:54:48.217297 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.217186    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-api-access-gt4cf" (OuterVolumeSpecName: "kube-api-access-gt4cf") pod "566fe303-800a-4a98-9a8b-f8fc3e2681e3" (UID: "566fe303-800a-4a98-9a8b-f8fc3e2681e3"). InnerVolumeSpecName "kube-api-access-gt4cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:54:48.315762 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.315736    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566fe303-800a-4a98-9a8b-f8fc3e2681e3-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:54:48.315762 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.315762    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:54:48.315907 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.315775    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt4cf\" (UniqueName: \"kubernetes.io/projected/566fe303-800a-4a98-9a8b-f8fc3e2681e3-kube-api-access-gt4cf\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:54:48.433999 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.433920    2575 generic.go:356] "Generic (PLEG): container finished" podID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerID="214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba" exitCode=0
Jul 28 15:54:48.433999 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.433975    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"
Jul 28 15:54:48.434427 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.434004    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" event={"ID":"566fe303-800a-4a98-9a8b-f8fc3e2681e3","Type":"ContainerDied","Data":"214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba"}
Jul 28 15:54:48.434427 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.434062    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59" event={"ID":"566fe303-800a-4a98-9a8b-f8fc3e2681e3","Type":"ContainerDied","Data":"68427a8617a8642891ed6f658bfda293dabefcdee84b74c989d94e972a6c3209"}
Jul 28 15:54:48.434427 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.434088    2575 scope.go:117] "RemoveContainer" containerID="6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09"
Jul 28 15:54:48.442333 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.442313    2575 scope.go:117] "RemoveContainer" containerID="214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba"
Jul 28 15:54:48.449686 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.449661    2575 scope.go:117] "RemoveContainer" containerID="6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09"
Jul 28 15:54:48.449932 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:54:48.449910    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09\": container with ID starting with 6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09 not found: ID does not exist" containerID="6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09"
Jul 28 15:54:48.450023 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.449938    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09"} err="failed to get container status \"6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09\": rpc error: code = NotFound desc = could not find container \"6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09\": container with ID starting with 6c3cdb650d8fafaae9e7bf618671d50bca620b3714fe0460cccf7f4e3f576c09 not found: ID does not exist"
Jul 28 15:54:48.450023 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.449954    2575 scope.go:117] "RemoveContainer" containerID="214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba"
Jul 28 15:54:48.450212 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:54:48.450190    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba\": container with ID starting with 214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba not found: ID does not exist" containerID="214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba"
Jul 28 15:54:48.450261 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.450220    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba"} err="failed to get container status \"214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba\": rpc error: code = NotFound desc = could not find container \"214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba\": container with ID starting with 214ecccb261207917ea1ffeefceaac87e8fda124beea4f403f774fa1c75e0dba not found: ID does not exist"
Jul 28 15:54:48.465613 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.465591    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"]
Jul 28 15:54:48.475444 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:48.475422    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-642bb-predictor-b88b6c887-hlk59"]
Jul 28 15:54:50.257242 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:50.257210    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" path="/var/lib/kubelet/pods/566fe303-800a-4a98-9a8b-f8fc3e2681e3/volumes"
Jul 28 15:54:52.289412 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:52.289374    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 15:54:52.431532 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:52.431499    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:54:52.432075 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:54:52.432041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 15:55:02.289339 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:02.289300    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:55:02.432707 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:02.432672    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 15:55:12.432071 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:12.431984    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 15:55:22.432996 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:22.432956    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 15:55:24.964823 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.964791    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"]
Jul 28 15:55:24.965448 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.965423    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kube-rbac-proxy"
Jul 28 15:55:24.965448 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.965449    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kube-rbac-proxy"
Jul 28 15:55:24.965604 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.965506    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container"
Jul 28 15:55:24.965604 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.965516    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container"
Jul 28 15:55:24.965604 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.965602    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kube-rbac-proxy"
Jul 28 15:55:24.965753 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.965617    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="566fe303-800a-4a98-9a8b-f8fc3e2681e3" containerName="kserve-container"
Jul 28 15:55:24.969176 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.969142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:24.976958 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.976933    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a87c3-predictor-serving-cert\""
Jul 28 15:55:24.977081 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.977057    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a87c3-kube-rbac-proxy-sar-config\""
Jul 28 15:55:24.987841 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:24.987815    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"]
Jul 28 15:55:25.055137 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.055110    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"]
Jul 28 15:55:25.055345 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.055310    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container" containerID="cri-o://c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39" gracePeriod=30
Jul 28 15:55:25.055482 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.055348    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kube-rbac-proxy" containerID="cri-o://d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792" gracePeriod=30
Jul 28 15:55:25.131888 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.131850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.132074 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.131955    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.132074 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.131979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gznk\" (UniqueName: \"kubernetes.io/projected/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-api-access-7gznk\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.233489 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.233413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.233489 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.233447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gznk\" (UniqueName: \"kubernetes.io/projected/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-api-access-7gznk\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.233711 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:25.233562    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-serving-cert: secret "success-200-isvc-a87c3-predictor-serving-cert" not found
Jul 28 15:55:25.233711 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.233582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.233711 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:25.233638    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls podName:e176455e-dbd6-4726-afc8-f4fb49e23800 nodeName:}" failed. No retries permitted until 2026-07-28 15:55:25.733619594 +0000 UTC m=+1361.983676577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls") pod "success-200-isvc-a87c3-predictor-6555844d55-tbqff" (UID: "e176455e-dbd6-4726-afc8-f4fb49e23800") : secret "success-200-isvc-a87c3-predictor-serving-cert" not found
Jul 28 15:55:25.234335 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.234315    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.250991 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.250966    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gznk\" (UniqueName: \"kubernetes.io/projected/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-api-access-7gznk\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.607191 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.607086    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerID="d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792" exitCode=2
Jul 28 15:55:25.607191 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.607149    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" event={"ID":"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163","Type":"ContainerDied","Data":"d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792"}
Jul 28 15:55:25.737671 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.737631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.740420 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.740392    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls\") pod \"success-200-isvc-a87c3-predictor-6555844d55-tbqff\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") " pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:25.882009 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:25.881921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:26.026895 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:26.026868    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"]
Jul 28 15:55:26.029030 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:55:26.029004    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode176455e_dbd6_4726_afc8_f4fb49e23800.slice/crio-eb4b44a46fd772eee793c743ec0f37089272eb5d7935659ab50512fa28ef043c WatchSource:0}: Error finding container eb4b44a46fd772eee793c743ec0f37089272eb5d7935659ab50512fa28ef043c: Status 404 returned error can't find the container with id eb4b44a46fd772eee793c743ec0f37089272eb5d7935659ab50512fa28ef043c
Jul 28 15:55:26.613965 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:26.613930    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" event={"ID":"e176455e-dbd6-4726-afc8-f4fb49e23800","Type":"ContainerStarted","Data":"947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96"}
Jul 28 15:55:26.614156 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:26.613994    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" event={"ID":"e176455e-dbd6-4726-afc8-f4fb49e23800","Type":"ContainerStarted","Data":"c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b"}
Jul 28 15:55:26.614156 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:26.614149    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:26.614310 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:26.614214    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:26.614310 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:26.614235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" event={"ID":"e176455e-dbd6-4726-afc8-f4fb49e23800","Type":"ContainerStarted","Data":"eb4b44a46fd772eee793c743ec0f37089272eb5d7935659ab50512fa28ef043c"}
Jul 28 15:55:26.615518 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:26.615493    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 15:55:26.653768 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:26.653724    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podStartSLOduration=2.6537105629999997 podStartE2EDuration="2.653710563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:55:26.646692054 +0000 UTC m=+1362.896749060" watchObservedRunningTime="2026-07-28 15:55:26.653710563 +0000 UTC m=+1362.903767569"
Jul 28 15:55:27.284794 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:27.284694    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 28 15:55:27.619024 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:27.618939    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 15:55:28.218031 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.218004    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:55:28.361513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.361433    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-rbac-proxy-sar-config\") pod \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") "
Jul 28 15:55:28.361513 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.361502    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-proxy-tls\") pod \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") "
Jul 28 15:55:28.361938 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.361554    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fscr2\" (UniqueName: \"kubernetes.io/projected/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-api-access-fscr2\") pod \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\" (UID: \"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163\") "
Jul 28 15:55:28.361938 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.361796    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" (UID: "c7ef5e1a-bb17-48f6-92ff-05e6a48c1163"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:55:28.363466 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.363435    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" (UID: "c7ef5e1a-bb17-48f6-92ff-05e6a48c1163"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:55:28.363579 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.363490    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-api-access-fscr2" (OuterVolumeSpecName: "kube-api-access-fscr2") pod "c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" (UID: "c7ef5e1a-bb17-48f6-92ff-05e6a48c1163"). InnerVolumeSpecName "kube-api-access-fscr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:55:28.462189 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.462137    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fscr2\" (UniqueName: \"kubernetes.io/projected/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-api-access-fscr2\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:55:28.462189 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.462185    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:55:28.462189 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.462196    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:55:28.624906 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.624820    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerID="c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39" exitCode=0
Jul 28 15:55:28.624906 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.624882    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"
Jul 28 15:55:28.625300 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.624911    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" event={"ID":"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163","Type":"ContainerDied","Data":"c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39"}
Jul 28 15:55:28.625300 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.624991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm" event={"ID":"c7ef5e1a-bb17-48f6-92ff-05e6a48c1163","Type":"ContainerDied","Data":"7f3f39d4fee8b85dc5c3ddba1bc40a4f57213af32e008b4895003f59b95300d6"}
Jul 28 15:55:28.625300 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.625018    2575 scope.go:117] "RemoveContainer" containerID="d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792"
Jul 28 15:55:28.634121 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.634103    2575 scope.go:117] "RemoveContainer" containerID="c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39"
Jul 28 15:55:28.641128 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.641112    2575 scope.go:117] "RemoveContainer" containerID="d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792"
Jul 28 15:55:28.641444 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:28.641410    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792\": container with ID starting with d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792 not found: ID does not exist" containerID="d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792"
Jul 28 15:55:28.641520 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.641451    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792"} err="failed to get container status \"d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792\": rpc error: code = NotFound desc = could not find container \"d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792\": container with ID starting with d95f2c0346e60efb4c48f34220b403d23351134a0bf8b1ac455a1df5424a6792 not found: ID does not exist"
Jul 28 15:55:28.641520 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.641467    2575 scope.go:117] "RemoveContainer" containerID="c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39"
Jul 28 15:55:28.641713 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:28.641693    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39\": container with ID starting with c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39 not found: ID does not exist" containerID="c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39"
Jul 28 15:55:28.641757 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.641721    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39"} err="failed to get container status \"c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39\": rpc error: code = NotFound desc = could not find container \"c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39\": container with ID starting with c61ff82f7d12d26eda4cd05d5bf37eba33a9089ededa13a04803e9fc70184a39 not found: ID does not exist"
Jul 28 15:55:28.656871 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.656852    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"]
Jul 28 15:55:28.664835 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:28.664816    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad35d-predictor-5d7d8b4bcc-wtmcm"]
Jul 28 15:55:30.257176 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:30.257140    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" path="/var/lib/kubelet/pods/c7ef5e1a-bb17-48f6-92ff-05e6a48c1163/volumes"
Jul 28 15:55:32.433809 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:32.433771    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:55:32.623779 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:32.623750    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:55:32.624359 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:32.624334    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 15:55:42.625181 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:42.625124    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 15:55:52.624483 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:52.624442    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 15:55:54.827439 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.827408    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"]
Jul 28 15:55:54.828359 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.827646    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container" containerID="cri-o://5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827" gracePeriod=30
Jul 28 15:55:54.828359 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.827669    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kube-rbac-proxy" containerID="cri-o://30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e" gracePeriod=30
Jul 28 15:55:54.937826 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.937795    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"]
Jul 28 15:55:54.938210 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.938196    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kube-rbac-proxy"
Jul 28 15:55:54.938210 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.938211    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kube-rbac-proxy"
Jul 28 15:55:54.938307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.938225    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container"
Jul 28 15:55:54.938307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.938231    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container"
Jul 28 15:55:54.938307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.938290    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kube-rbac-proxy"
Jul 28 15:55:54.938307 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.938298    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7ef5e1a-bb17-48f6-92ff-05e6a48c1163" containerName="kserve-container"
Jul 28 15:55:54.942218 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.942202    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:54.945384 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.945366    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-50d42-predictor-serving-cert\""
Jul 28 15:55:54.945628 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.945605    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-50d42-kube-rbac-proxy-sar-config\""
Jul 28 15:55:54.955357 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.955335    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"]
Jul 28 15:55:54.990984 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.990951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:54.990984 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.990993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b525d\" (UniqueName: \"kubernetes.io/projected/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-api-access-b525d\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:54.991233 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:54.991013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.092634 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.092536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.092634 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.092585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b525d\" (UniqueName: \"kubernetes.io/projected/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-api-access-b525d\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.092861 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.092687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.092861 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:55.092704    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-50d42-predictor-serving-cert: secret "success-200-isvc-50d42-predictor-serving-cert" not found
Jul 28 15:55:55.092861 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:55.092773    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls podName:eaca4477-e19f-4dd6-9048-5ea6a3f63b40 nodeName:}" failed. No retries permitted until 2026-07-28 15:55:55.592752921 +0000 UTC m=+1391.842809912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls") pod "success-200-isvc-50d42-predictor-554687cbf-4nw4f" (UID: "eaca4477-e19f-4dd6-9048-5ea6a3f63b40") : secret "success-200-isvc-50d42-predictor-serving-cert" not found
Jul 28 15:55:55.093448 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.093424    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.111322 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.111299    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b525d\" (UniqueName: \"kubernetes.io/projected/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-api-access-b525d\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.597160 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.597121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.599504 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.599479    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls\") pod \"success-200-isvc-50d42-predictor-554687cbf-4nw4f\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") " pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.752689 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.752655    2575 generic.go:356] "Generic (PLEG): container finished" podID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerID="30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e" exitCode=2
Jul 28 15:55:55.752853 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.752697    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" event={"ID":"7c7c7959-59bd-492e-87f4-1099cf7b46b8","Type":"ContainerDied","Data":"30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e"}
Jul 28 15:55:55.852771 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.852678    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:55.995639 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:55.995612    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"]
Jul 28 15:55:55.998138 ip-10-0-143-228 kubenswrapper[2575]: W0728 15:55:55.998111    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaca4477_e19f_4dd6_9048_5ea6a3f63b40.slice/crio-013df5fa69115688a2ecd3f83cf1ada6c189b0391b201ab37ea2d1b206e72012 WatchSource:0}: Error finding container 013df5fa69115688a2ecd3f83cf1ada6c189b0391b201ab37ea2d1b206e72012: Status 404 returned error can't find the container with id 013df5fa69115688a2ecd3f83cf1ada6c189b0391b201ab37ea2d1b206e72012
Jul 28 15:55:56.759604 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:56.759570    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" event={"ID":"eaca4477-e19f-4dd6-9048-5ea6a3f63b40","Type":"ContainerStarted","Data":"8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a"}
Jul 28 15:55:56.759779 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:56.759625    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" event={"ID":"eaca4477-e19f-4dd6-9048-5ea6a3f63b40","Type":"ContainerStarted","Data":"bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0"}
Jul 28 15:55:56.759779 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:56.759645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" event={"ID":"eaca4477-e19f-4dd6-9048-5ea6a3f63b40","Type":"ContainerStarted","Data":"013df5fa69115688a2ecd3f83cf1ada6c189b0391b201ab37ea2d1b206e72012"}
Jul 28 15:55:56.759869 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:56.759824    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:56.759980 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:56.759957    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:55:56.761248 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:56.761221    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 15:55:56.783132 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:56.783072    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podStartSLOduration=2.783056844 podStartE2EDuration="2.783056844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:55:56.781618951 +0000 UTC m=+1393.031675957" watchObservedRunningTime="2026-07-28 15:55:56.783056844 +0000 UTC m=+1393.033113851"
Jul 28 15:55:57.428465 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:57.428421    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 15:55:57.763989 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:57.763955    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 15:55:57.925441 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:57.925406    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7c7959_59bd_492e_87f4_1099cf7b46b8.slice/crio-conmon-5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 15:55:58.069591 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.069567    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:55:58.219526 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.219489    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls\") pod \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") "
Jul 28 15:55:58.219694 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.219544    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdkns\" (UniqueName: \"kubernetes.io/projected/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-api-access-zdkns\") pod \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") "
Jul 28 15:55:58.219694 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.219583    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-rbac-proxy-sar-config\") pod \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\" (UID: \"7c7c7959-59bd-492e-87f4-1099cf7b46b8\") "
Jul 28 15:55:58.219943 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.219923    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c7c7959-59bd-492e-87f4-1099cf7b46b8" (UID: "7c7c7959-59bd-492e-87f4-1099cf7b46b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:55:58.221602 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.221576    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c7c7959-59bd-492e-87f4-1099cf7b46b8" (UID: "7c7c7959-59bd-492e-87f4-1099cf7b46b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:55:58.221709 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.221635    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-api-access-zdkns" (OuterVolumeSpecName: "kube-api-access-zdkns") pod "7c7c7959-59bd-492e-87f4-1099cf7b46b8" (UID: "7c7c7959-59bd-492e-87f4-1099cf7b46b8"). InnerVolumeSpecName "kube-api-access-zdkns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:55:58.320520 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.320445    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkns\" (UniqueName: \"kubernetes.io/projected/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-api-access-zdkns\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:55:58.320520 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.320474    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7c7959-59bd-492e-87f4-1099cf7b46b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:55:58.320520 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.320487    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7c7959-59bd-492e-87f4-1099cf7b46b8-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 15:55:58.770537 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.770502    2575 generic.go:356] "Generic (PLEG): container finished" podID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerID="5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827" exitCode=0
Jul 28 15:55:58.771010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.770559    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"
Jul 28 15:55:58.771010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.770582    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" event={"ID":"7c7c7959-59bd-492e-87f4-1099cf7b46b8","Type":"ContainerDied","Data":"5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827"}
Jul 28 15:55:58.771010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.770634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq" event={"ID":"7c7c7959-59bd-492e-87f4-1099cf7b46b8","Type":"ContainerDied","Data":"05b4781bfc5aef9cabf1cce060a3f6130b2d1b786af51e05f1bf23a5251e6f34"}
Jul 28 15:55:58.771010 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.770661    2575 scope.go:117] "RemoveContainer" containerID="30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e"
Jul 28 15:55:58.778934 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.778906    2575 scope.go:117] "RemoveContainer" containerID="5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827"
Jul 28 15:55:58.788060 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.788042    2575 scope.go:117] "RemoveContainer" containerID="30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e"
Jul 28 15:55:58.788369 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:58.788347    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e\": container with ID starting with 30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e not found: ID does not exist" containerID="30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e"
Jul 28 15:55:58.788480 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.788376    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e"} err="failed to get container status \"30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e\": rpc error: code = NotFound desc = could not find container \"30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e\": container with ID starting with 30ad2e1afe758a7b5a32415c60fd042c257f07cdd30ac37274edb0af1a0aef6e not found: ID does not exist"
Jul 28 15:55:58.788480 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.788392    2575 scope.go:117] "RemoveContainer" containerID="5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827"
Jul 28 15:55:58.788634 ip-10-0-143-228 kubenswrapper[2575]: E0728 15:55:58.788618    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827\": container with ID starting with 5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827 not found: ID does not exist" containerID="5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827"
Jul 28 15:55:58.788676 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.788642    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827"} err="failed to get container status \"5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827\": rpc error: code = NotFound desc = could not find container \"5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827\": container with ID starting with 5d0bc135d09c17dc059925fa8a4eb6cc0007025e209b1ca802069129ade24827 not found: ID does not exist"
Jul 28 15:55:58.792594 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.792571    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"]
Jul 28 15:55:58.799207 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:55:58.799187    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12202-predictor-f74599657-2zkrq"]
Jul 28 15:56:00.256986 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:00.256948    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" path="/var/lib/kubelet/pods/7c7c7959-59bd-492e-87f4-1099cf7b46b8/volumes"
Jul 28 15:56:02.624374 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:02.624335    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 15:56:02.769068 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:02.769036    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:56:02.769529 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:02.769502    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 15:56:12.625346 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:12.625313    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 15:56:12.769982 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:12.769944    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 15:56:22.770422 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:22.770371    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 15:56:32.770077 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:32.770039    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 15:56:42.771240 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:56:42.771151    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 15:57:44.251647 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:57:44.251616    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:57:44.252443 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:57:44.252423    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 15:57:44.262429 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:57:44.262410    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 15:57:44.263095 ip-10-0-143-228 kubenswrapper[2575]: I0728 15:57:44.263075    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:02:44.285654 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:02:44.285623    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:02:44.287933 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:02:44.287904    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:02:44.291672 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:02:44.291651    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:02:44.294034 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:02:44.294018    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:04:39.652428 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.652345    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"]
Jul 28 16:04:39.652987 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.652567    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container" containerID="cri-o://c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b" gracePeriod=30
Jul 28 16:04:39.652987 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.652627    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kube-rbac-proxy" containerID="cri-o://947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96" gracePeriod=30
Jul 28 16:04:39.732299 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.732262    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"]
Jul 28 16:04:39.732807 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.732793    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kube-rbac-proxy"
Jul 28 16:04:39.732866 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.732810    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kube-rbac-proxy"
Jul 28 16:04:39.732866 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.732851    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container"
Jul 28 16:04:39.732866 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.732860    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container"
Jul 28 16:04:39.732975 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.732939    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kserve-container"
Jul 28 16:04:39.732975 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.732950    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7c7959-59bd-492e-87f4-1099cf7b46b8" containerName="kube-rbac-proxy"
Jul 28 16:04:39.736425 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.736408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:39.739099 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.739075    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b87a7-predictor-serving-cert\""
Jul 28 16:04:39.739246 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.739221    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b87a7-kube-rbac-proxy-sar-config\""
Jul 28 16:04:39.749231 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.749211    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"]
Jul 28 16:04:39.894616 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.894584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:39.894787 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.894630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vl4w\" (UniqueName: \"kubernetes.io/projected/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-api-access-5vl4w\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:39.894787 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.894758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:39.995962 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.995926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:39.996149 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.995980    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:39.996149 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.996003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vl4w\" (UniqueName: \"kubernetes.io/projected/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-api-access-5vl4w\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:39.996149 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:04:39.996093    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-serving-cert: secret "success-200-isvc-b87a7-predictor-serving-cert" not found
Jul 28 16:04:39.996320 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:04:39.996211    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls podName:d4bb668c-0cfe-49e7-9ddd-1c37b7458369 nodeName:}" failed. No retries permitted until 2026-07-28 16:04:40.496157994 +0000 UTC m=+1916.746214980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls") pod "success-200-isvc-b87a7-predictor-7865b79897-nxq5c" (UID: "d4bb668c-0cfe-49e7-9ddd-1c37b7458369") : secret "success-200-isvc-b87a7-predictor-serving-cert" not found
Jul 28 16:04:39.996764 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:39.996744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:40.004934 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:40.004904    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vl4w\" (UniqueName: \"kubernetes.io/projected/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-api-access-5vl4w\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:40.194437 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:40.194404    2575 generic.go:356] "Generic (PLEG): container finished" podID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerID="947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96" exitCode=2
Jul 28 16:04:40.194612 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:40.194474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" event={"ID":"e176455e-dbd6-4726-afc8-f4fb49e23800","Type":"ContainerDied","Data":"947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96"}
Jul 28 16:04:40.499817 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:40.499785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:40.502159 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:40.502125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls\") pod \"success-200-isvc-b87a7-predictor-7865b79897-nxq5c\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") " pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:40.646636 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:40.646591    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:40.775064 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:40.775040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"]
Jul 28 16:04:40.776813 ip-10-0-143-228 kubenswrapper[2575]: W0728 16:04:40.776784    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bb668c_0cfe_49e7_9ddd_1c37b7458369.slice/crio-e0051e73b2103c5886c3500950cfada2fe0a56f957ee88b14fc6f0a81f41c87e WatchSource:0}: Error finding container e0051e73b2103c5886c3500950cfada2fe0a56f957ee88b14fc6f0a81f41c87e: Status 404 returned error can't find the container with id e0051e73b2103c5886c3500950cfada2fe0a56f957ee88b14fc6f0a81f41c87e
Jul 28 16:04:41.201962 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:41.201925    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" event={"ID":"d4bb668c-0cfe-49e7-9ddd-1c37b7458369","Type":"ContainerStarted","Data":"13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136"}
Jul 28 16:04:41.202136 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:41.202007    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" event={"ID":"d4bb668c-0cfe-49e7-9ddd-1c37b7458369","Type":"ContainerStarted","Data":"68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1"}
Jul 28 16:04:41.202136 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:41.202034    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:41.202136 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:41.202056    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:41.202136 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:41.202073    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" event={"ID":"d4bb668c-0cfe-49e7-9ddd-1c37b7458369","Type":"ContainerStarted","Data":"e0051e73b2103c5886c3500950cfada2fe0a56f957ee88b14fc6f0a81f41c87e"}
Jul 28 16:04:41.203535 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:41.203505    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 16:04:41.225138 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:41.225099    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podStartSLOduration=2.225086181 podStartE2EDuration="2.225086181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:41.223556578 +0000 UTC m=+1917.473613584" watchObservedRunningTime="2026-07-28 16:04:41.225086181 +0000 UTC m=+1917.475143185"
Jul 28 16:04:42.207074 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:42.207030    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 16:04:42.619491 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:42.619400    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 28 16:04:42.624695 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:42.624670    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 16:04:42.896305 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:42.896281    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 16:04:43.020051 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.020009    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gznk\" (UniqueName: \"kubernetes.io/projected/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-api-access-7gznk\") pod \"e176455e-dbd6-4726-afc8-f4fb49e23800\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") "
Jul 28 16:04:43.020261 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.020118    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-rbac-proxy-sar-config\") pod \"e176455e-dbd6-4726-afc8-f4fb49e23800\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") "
Jul 28 16:04:43.020261 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.020194    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls\") pod \"e176455e-dbd6-4726-afc8-f4fb49e23800\" (UID: \"e176455e-dbd6-4726-afc8-f4fb49e23800\") "
Jul 28 16:04:43.020503 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.020478    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e176455e-dbd6-4726-afc8-f4fb49e23800" (UID: "e176455e-dbd6-4726-afc8-f4fb49e23800"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:43.022231 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.022201    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e176455e-dbd6-4726-afc8-f4fb49e23800" (UID: "e176455e-dbd6-4726-afc8-f4fb49e23800"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:43.022345 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.022239    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-api-access-7gznk" (OuterVolumeSpecName: "kube-api-access-7gznk") pod "e176455e-dbd6-4726-afc8-f4fb49e23800" (UID: "e176455e-dbd6-4726-afc8-f4fb49e23800"). InnerVolumeSpecName "kube-api-access-7gznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:43.121448 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.121416    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:04:43.121448 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.121441    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e176455e-dbd6-4726-afc8-f4fb49e23800-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:04:43.121448 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.121451    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gznk\" (UniqueName: \"kubernetes.io/projected/e176455e-dbd6-4726-afc8-f4fb49e23800-kube-api-access-7gznk\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:04:43.213305 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.213269    2575 generic.go:356] "Generic (PLEG): container finished" podID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerID="c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b" exitCode=0
Jul 28 16:04:43.213670 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.213319    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"
Jul 28 16:04:43.213670 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.213323    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" event={"ID":"e176455e-dbd6-4726-afc8-f4fb49e23800","Type":"ContainerDied","Data":"c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b"}
Jul 28 16:04:43.213670 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.213383    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff" event={"ID":"e176455e-dbd6-4726-afc8-f4fb49e23800","Type":"ContainerDied","Data":"eb4b44a46fd772eee793c743ec0f37089272eb5d7935659ab50512fa28ef043c"}
Jul 28 16:04:43.213670 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.213424    2575 scope.go:117] "RemoveContainer" containerID="947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96"
Jul 28 16:04:43.222121 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.222101    2575 scope.go:117] "RemoveContainer" containerID="c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b"
Jul 28 16:04:43.229739 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.229723    2575 scope.go:117] "RemoveContainer" containerID="947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96"
Jul 28 16:04:43.229990 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:04:43.229972    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96\": container with ID starting with 947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96 not found: ID does not exist" containerID="947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96"
Jul 28 16:04:43.230060 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.229999    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96"} err="failed to get container status \"947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96\": rpc error: code = NotFound desc = could not find container \"947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96\": container with ID starting with 947dbc1dd77e1c6b74aeef7b19a3c242e78e3417195514f4d22b457d8ae00a96 not found: ID does not exist"
Jul 28 16:04:43.230060 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.230016    2575 scope.go:117] "RemoveContainer" containerID="c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b"
Jul 28 16:04:43.230262 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:04:43.230243    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b\": container with ID starting with c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b not found: ID does not exist" containerID="c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b"
Jul 28 16:04:43.230359 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.230263    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b"} err="failed to get container status \"c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b\": rpc error: code = NotFound desc = could not find container \"c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b\": container with ID starting with c555402a40fa49353c7d8c33c48f3afe2e90da5d486be7947ae4251dc410594b not found: ID does not exist"
Jul 28 16:04:43.236549 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.236527    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"]
Jul 28 16:04:43.241240 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:43.241221    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a87c3-predictor-6555844d55-tbqff"]
Jul 28 16:04:44.256508 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:44.256477    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" path="/var/lib/kubelet/pods/e176455e-dbd6-4726-afc8-f4fb49e23800/volumes"
Jul 28 16:04:47.211534 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:47.211503    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:04:47.212041 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:47.211939    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 16:04:57.212766 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:04:57.212727    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 16:05:07.212557 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:07.212516    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 16:05:09.731907 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.731866    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"]
Jul 28 16:05:09.732406 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.732199    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kube-rbac-proxy" containerID="cri-o://8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a" gracePeriod=30
Jul 28 16:05:09.732406 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.732159    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container" containerID="cri-o://bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0" gracePeriod=30
Jul 28 16:05:09.791266 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.791233    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"]
Jul 28 16:05:09.791859 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.791828    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container"
Jul 28 16:05:09.791859 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.791848    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container"
Jul 28 16:05:09.792060 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.791920    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kube-rbac-proxy"
Jul 28 16:05:09.792060 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.791931    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kube-rbac-proxy"
Jul 28 16:05:09.792060 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.792023    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kube-rbac-proxy"
Jul 28 16:05:09.792060 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.792035    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e176455e-dbd6-4726-afc8-f4fb49e23800" containerName="kserve-container"
Jul 28 16:05:09.795575 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.795556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:09.798412 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.798387    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bd785-predictor-serving-cert\""
Jul 28 16:05:09.798514 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.798441    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bd785-kube-rbac-proxy-sar-config\""
Jul 28 16:05:09.805236 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.805213    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"]
Jul 28 16:05:09.861378 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.861345    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:09.861610 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.861472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7g9\" (UniqueName: \"kubernetes.io/projected/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-api-access-lk7g9\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:09.861610 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.861508    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:09.962346 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.962304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7g9\" (UniqueName: \"kubernetes.io/projected/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-api-access-lk7g9\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:09.962563 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.962356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:09.962563 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.962393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:09.962703 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:09.962557    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-bd785-predictor-serving-cert: secret "success-200-isvc-bd785-predictor-serving-cert" not found
Jul 28 16:05:09.962703 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:09.962626    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls podName:bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd nodeName:}" failed. No retries permitted until 2026-07-28 16:05:10.462606244 +0000 UTC m=+1946.712663242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls") pod "success-200-isvc-bd785-predictor-989fd784c-ndhgf" (UID: "bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd") : secret "success-200-isvc-bd785-predictor-serving-cert" not found
Jul 28 16:05:09.963015 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.962997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:09.973363 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:09.973339    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7g9\" (UniqueName: \"kubernetes.io/projected/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-api-access-lk7g9\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:10.337874 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:10.337839    2575 generic.go:356] "Generic (PLEG): container finished" podID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerID="8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a" exitCode=2
Jul 28 16:05:10.338039 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:10.337879    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" event={"ID":"eaca4477-e19f-4dd6-9048-5ea6a3f63b40","Type":"ContainerDied","Data":"8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a"}
Jul 28 16:05:10.468049 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:10.468011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:10.468285 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:10.468205    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-bd785-predictor-serving-cert: secret "success-200-isvc-bd785-predictor-serving-cert" not found
Jul 28 16:05:10.468285 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:10.468286    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls podName:bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd nodeName:}" failed. No retries permitted until 2026-07-28 16:05:11.468265693 +0000 UTC m=+1947.718322679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls") pod "success-200-isvc-bd785-predictor-989fd784c-ndhgf" (UID: "bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd") : secret "success-200-isvc-bd785-predictor-serving-cert" not found
Jul 28 16:05:11.476387 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:11.476343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:11.478899 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:11.478873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls\") pod \"success-200-isvc-bd785-predictor-989fd784c-ndhgf\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") " pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:11.608060 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:11.608024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:11.737794 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:11.737767    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"]
Jul 28 16:05:11.739521 ip-10-0-143-228 kubenswrapper[2575]: W0728 16:05:11.739498    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce3f050_9c6b_4a4f_a217_cbc4ab0fd4dd.slice/crio-144cafa1e65f3c915a08c95ba44a62498cb345c2bceaef084cf342107852a283 WatchSource:0}: Error finding container 144cafa1e65f3c915a08c95ba44a62498cb345c2bceaef084cf342107852a283: Status 404 returned error can't find the container with id 144cafa1e65f3c915a08c95ba44a62498cb345c2bceaef084cf342107852a283
Jul 28 16:05:12.347493 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:12.347453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" event={"ID":"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd","Type":"ContainerStarted","Data":"ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d"}
Jul 28 16:05:12.347689 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:12.347516    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" event={"ID":"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd","Type":"ContainerStarted","Data":"c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051"}
Jul 28 16:05:12.347689 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:12.347539    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" event={"ID":"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd","Type":"ContainerStarted","Data":"144cafa1e65f3c915a08c95ba44a62498cb345c2bceaef084cf342107852a283"}
Jul 28 16:05:12.347689 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:12.347587    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:12.368394 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:12.368346    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podStartSLOduration=3.368331951 podStartE2EDuration="3.368331951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:12.367905811 +0000 UTC m=+1948.617962839" watchObservedRunningTime="2026-07-28 16:05:12.368331951 +0000 UTC m=+1948.618388956"
Jul 28 16:05:12.764738 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:12.764692    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 16:05:12.770125 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:12.770092    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 16:05:13.352154 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.352117    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:13.353393 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.353367    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 16:05:13.679368 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.679343    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 16:05:13.795692 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.795654    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b525d\" (UniqueName: \"kubernetes.io/projected/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-api-access-b525d\") pod \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") "
Jul 28 16:05:13.796152 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.795730    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-rbac-proxy-sar-config\") pod \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") "
Jul 28 16:05:13.796152 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.795777    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls\") pod \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\" (UID: \"eaca4477-e19f-4dd6-9048-5ea6a3f63b40\") "
Jul 28 16:05:13.796152 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.796025    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eaca4477-e19f-4dd6-9048-5ea6a3f63b40" (UID: "eaca4477-e19f-4dd6-9048-5ea6a3f63b40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:13.797866 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.797838    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-api-access-b525d" (OuterVolumeSpecName: "kube-api-access-b525d") pod "eaca4477-e19f-4dd6-9048-5ea6a3f63b40" (UID: "eaca4477-e19f-4dd6-9048-5ea6a3f63b40"). InnerVolumeSpecName "kube-api-access-b525d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:13.797866 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.797860    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eaca4477-e19f-4dd6-9048-5ea6a3f63b40" (UID: "eaca4477-e19f-4dd6-9048-5ea6a3f63b40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:13.897308 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.897280    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:05:13.897308 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.897304    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b525d\" (UniqueName: \"kubernetes.io/projected/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-api-access-b525d\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:05:13.897466 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:13.897316    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaca4477-e19f-4dd6-9048-5ea6a3f63b40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:05:14.357533 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.357501    2575 generic.go:356] "Generic (PLEG): container finished" podID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerID="bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0" exitCode=0
Jul 28 16:05:14.357697 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.357555    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"
Jul 28 16:05:14.357697 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.357584    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" event={"ID":"eaca4477-e19f-4dd6-9048-5ea6a3f63b40","Type":"ContainerDied","Data":"bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0"}
Jul 28 16:05:14.357697 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.357638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f" event={"ID":"eaca4477-e19f-4dd6-9048-5ea6a3f63b40","Type":"ContainerDied","Data":"013df5fa69115688a2ecd3f83cf1ada6c189b0391b201ab37ea2d1b206e72012"}
Jul 28 16:05:14.357697 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.357680    2575 scope.go:117] "RemoveContainer" containerID="8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a"
Jul 28 16:05:14.358126 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.358100    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 16:05:14.365771 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.365756    2575 scope.go:117] "RemoveContainer" containerID="bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0"
Jul 28 16:05:14.373233 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.373219    2575 scope.go:117] "RemoveContainer" containerID="8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a"
Jul 28 16:05:14.373477 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:14.373460    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a\": container with ID starting with 8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a not found: ID does not exist" containerID="8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a"
Jul 28 16:05:14.373521 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.373493    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a"} err="failed to get container status \"8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a\": rpc error: code = NotFound desc = could not find container \"8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a\": container with ID starting with 8c06c22f6d40c8c4cd043df663d0105b646851944d2bcb1ccec4e64584ca643a not found: ID does not exist"
Jul 28 16:05:14.373521 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.373511    2575 scope.go:117] "RemoveContainer" containerID="bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0"
Jul 28 16:05:14.373746 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:14.373729    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0\": container with ID starting with bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0 not found: ID does not exist" containerID="bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0"
Jul 28 16:05:14.373795 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.373759    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0"} err="failed to get container status \"bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0\": rpc error: code = NotFound desc = could not find container \"bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0\": container with ID starting with bf51a1466282e33513562dc537c4e5dc7072f693469e1fca5744f0c7a48b5ca0 not found: ID does not exist"
Jul 28 16:05:14.382031 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.382005    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"]
Jul 28 16:05:14.386208 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:14.386185    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-50d42-predictor-554687cbf-4nw4f"]
Jul 28 16:05:16.256351 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:16.256317    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" path="/var/lib/kubelet/pods/eaca4477-e19f-4dd6-9048-5ea6a3f63b40/volumes"
Jul 28 16:05:17.212630 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:17.212589    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 16:05:19.362834 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:19.362804    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:05:19.363339 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:19.363222    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 16:05:27.213300 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:27.213272    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:05:29.363352 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:29.363313    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 16:05:39.363447 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:39.363362    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 16:05:49.364000 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:49.363961    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 16:05:49.977498 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:49.977461    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"]
Jul 28 16:05:49.977734 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:49.977710    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container" containerID="cri-o://68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1" gracePeriod=30
Jul 28 16:05:49.977802 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:49.977756    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kube-rbac-proxy" containerID="cri-o://13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136" gracePeriod=30
Jul 28 16:05:50.061427 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.061393    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"]
Jul 28 16:05:50.061949 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.061933    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kube-rbac-proxy"
Jul 28 16:05:50.061949 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.061950    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kube-rbac-proxy"
Jul 28 16:05:50.062039 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.061971    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container"
Jul 28 16:05:50.062039 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.061979    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container"
Jul 28 16:05:50.062106 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.062079    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kube-rbac-proxy"
Jul 28 16:05:50.062106 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.062092    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaca4477-e19f-4dd6-9048-5ea6a3f63b40" containerName="kserve-container"
Jul 28 16:05:50.065669 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.065649    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.068539 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.068520    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dd6ec-predictor-serving-cert\""
Jul 28 16:05:50.068619 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.068526    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dd6ec-kube-rbac-proxy-sar-config\""
Jul 28 16:05:50.073421 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.073354    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"]
Jul 28 16:05:50.102667 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.102642    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcxn\" (UniqueName: \"kubernetes.io/projected/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-api-access-jvcxn\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.102803 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.102675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.102803 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.102706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.203985 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.203951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcxn\" (UniqueName: \"kubernetes.io/projected/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-api-access-jvcxn\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.203985 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.203995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.204336 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.204028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.204336 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:50.204186    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-serving-cert: secret "success-200-isvc-dd6ec-predictor-serving-cert" not found
Jul 28 16:05:50.204336 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:50.204265    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls podName:575ece81-6bb8-4086-813f-d28b7c4eab37 nodeName:}" failed. No retries permitted until 2026-07-28 16:05:50.704248203 +0000 UTC m=+1986.954305191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls") pod "success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" (UID: "575ece81-6bb8-4086-813f-d28b7c4eab37") : secret "success-200-isvc-dd6ec-predictor-serving-cert" not found
Jul 28 16:05:50.204767 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.204747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.213342 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.213317    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcxn\" (UniqueName: \"kubernetes.io/projected/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-api-access-jvcxn\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.524498 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.524466    2575 generic.go:356] "Generic (PLEG): container finished" podID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerID="13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136" exitCode=2
Jul 28 16:05:50.524891 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.524532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" event={"ID":"d4bb668c-0cfe-49e7-9ddd-1c37b7458369","Type":"ContainerDied","Data":"13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136"}
Jul 28 16:05:50.708817 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.708784    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.711090 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.711061    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls\") pod \"success-200-isvc-dd6ec-predictor-6948f85895-wp4q7\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") " pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:50.977895 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:50.977858    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:51.110620 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:51.110595    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"]
Jul 28 16:05:51.111904 ip-10-0-143-228 kubenswrapper[2575]: W0728 16:05:51.111881    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575ece81_6bb8_4086_813f_d28b7c4eab37.slice/crio-966220e09a2060b63e060bebef201dcad6fec306fb7260f6d1897c53ca858f0c WatchSource:0}: Error finding container 966220e09a2060b63e060bebef201dcad6fec306fb7260f6d1897c53ca858f0c: Status 404 returned error can't find the container with id 966220e09a2060b63e060bebef201dcad6fec306fb7260f6d1897c53ca858f0c
Jul 28 16:05:51.531269 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:51.531230    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" event={"ID":"575ece81-6bb8-4086-813f-d28b7c4eab37","Type":"ContainerStarted","Data":"820c3557ddf91f67c781ddc9210d094a19c251f3d3001747d32ca3253aaf22ed"}
Jul 28 16:05:51.531643 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:51.531303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" event={"ID":"575ece81-6bb8-4086-813f-d28b7c4eab37","Type":"ContainerStarted","Data":"524ce9829651df63f20dc4b1ab9fec00bb947cc525568a12b5ace56426be07b1"}
Jul 28 16:05:51.531643 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:51.531335    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:51.531643 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:51.531365    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" event={"ID":"575ece81-6bb8-4086-813f-d28b7c4eab37","Type":"ContainerStarted","Data":"966220e09a2060b63e060bebef201dcad6fec306fb7260f6d1897c53ca858f0c"}
Jul 28 16:05:51.554621 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:51.554575    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podStartSLOduration=1.554561542 podStartE2EDuration="1.554561542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:51.552872589 +0000 UTC m=+1987.802929620" watchObservedRunningTime="2026-07-28 16:05:51.554561542 +0000 UTC m=+1987.804618525"
Jul 28 16:05:52.207408 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:52.207358    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 28 16:05:52.536771 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:52.536688    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:52.538074 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:52.538045    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 16:05:53.325348 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.325323    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:05:53.432749 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.432643    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vl4w\" (UniqueName: \"kubernetes.io/projected/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-api-access-5vl4w\") pod \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") "
Jul 28 16:05:53.432749 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.432708    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-rbac-proxy-sar-config\") pod \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") "
Jul 28 16:05:53.432749 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.432747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls\") pod \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\" (UID: \"d4bb668c-0cfe-49e7-9ddd-1c37b7458369\") "
Jul 28 16:05:53.433205 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.433147    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4bb668c-0cfe-49e7-9ddd-1c37b7458369" (UID: "d4bb668c-0cfe-49e7-9ddd-1c37b7458369"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:53.434783 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.434753    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-api-access-5vl4w" (OuterVolumeSpecName: "kube-api-access-5vl4w") pod "d4bb668c-0cfe-49e7-9ddd-1c37b7458369" (UID: "d4bb668c-0cfe-49e7-9ddd-1c37b7458369"). InnerVolumeSpecName "kube-api-access-5vl4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:53.434866 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.434798    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4bb668c-0cfe-49e7-9ddd-1c37b7458369" (UID: "d4bb668c-0cfe-49e7-9ddd-1c37b7458369"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:53.533357 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.533320    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vl4w\" (UniqueName: \"kubernetes.io/projected/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-api-access-5vl4w\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:05:53.533357 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.533349    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:05:53.533357 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.533360    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4bb668c-0cfe-49e7-9ddd-1c37b7458369-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:05:53.542750 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.542715    2575 generic.go:356] "Generic (PLEG): container finished" podID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerID="68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1" exitCode=0
Jul 28 16:05:53.543101 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.542791    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"
Jul 28 16:05:53.543101 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.542797    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" event={"ID":"d4bb668c-0cfe-49e7-9ddd-1c37b7458369","Type":"ContainerDied","Data":"68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1"}
Jul 28 16:05:53.543101 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.542901    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c" event={"ID":"d4bb668c-0cfe-49e7-9ddd-1c37b7458369","Type":"ContainerDied","Data":"e0051e73b2103c5886c3500950cfada2fe0a56f957ee88b14fc6f0a81f41c87e"}
Jul 28 16:05:53.543101 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.542929    2575 scope.go:117] "RemoveContainer" containerID="13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136"
Jul 28 16:05:53.543634 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.543603    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 16:05:53.551990 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.551975    2575 scope.go:117] "RemoveContainer" containerID="68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1"
Jul 28 16:05:53.559540 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.559523    2575 scope.go:117] "RemoveContainer" containerID="13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136"
Jul 28 16:05:53.559788 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:53.559768    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136\": container with ID starting with 13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136 not found: ID does not exist" containerID="13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136"
Jul 28 16:05:53.559834 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.559797    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136"} err="failed to get container status \"13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136\": rpc error: code = NotFound desc = could not find container \"13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136\": container with ID starting with 13050244424b22bbb1bfc0126a017486707fe2c897c02802e0d6a052808ed136 not found: ID does not exist"
Jul 28 16:05:53.559834 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.559812    2575 scope.go:117] "RemoveContainer" containerID="68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1"
Jul 28 16:05:53.560034 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:05:53.560016    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1\": container with ID starting with 68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1 not found: ID does not exist" containerID="68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1"
Jul 28 16:05:53.560088 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.560042    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1"} err="failed to get container status \"68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1\": rpc error: code = NotFound desc = could not find container \"68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1\": container with ID starting with 68110fc04bcf6ed972951972e45c5a076190b65f19b12e1244ea06dc8ec968a1 not found: ID does not exist"
Jul 28 16:05:53.566533 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.566506    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"]
Jul 28 16:05:53.570690 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:53.570669    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b87a7-predictor-7865b79897-nxq5c"]
Jul 28 16:05:54.257001 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:54.256969    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" path="/var/lib/kubelet/pods/d4bb668c-0cfe-49e7-9ddd-1c37b7458369/volumes"
Jul 28 16:05:58.548213 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:58.548186    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:05:58.548734 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:58.548706    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 16:05:59.363732 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:05:59.363698    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:06:08.548861 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:06:08.548819    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 16:06:18.548751 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:06:18.548713    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 16:06:28.548964 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:06:28.548919    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 16:06:38.549329 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:06:38.549291    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:07:44.316338 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:07:44.316301    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:07:44.320764 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:07:44.320741    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:07:44.322905 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:07:44.322886    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:07:44.327398 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:07:44.327383    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:12:44.348774 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:12:44.348743    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:12:44.353650 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:12:44.353628    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:12:44.355044 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:12:44.355025    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:12:44.359633 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:12:44.359614    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:15:04.832810 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:04.832774    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"]
Jul 28 16:15:04.833451 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:04.833032    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container" containerID="cri-o://524ce9829651df63f20dc4b1ab9fec00bb947cc525568a12b5ace56426be07b1" gracePeriod=30
Jul 28 16:15:04.833451 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:04.833089    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kube-rbac-proxy" containerID="cri-o://820c3557ddf91f67c781ddc9210d094a19c251f3d3001747d32ca3253aaf22ed" gracePeriod=30
Jul 28 16:15:05.079238 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:05.079201    2575 generic.go:356] "Generic (PLEG): container finished" podID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerID="820c3557ddf91f67c781ddc9210d094a19c251f3d3001747d32ca3253aaf22ed" exitCode=2
Jul 28 16:15:05.079448 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:05.079261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" event={"ID":"575ece81-6bb8-4086-813f-d28b7c4eab37","Type":"ContainerDied","Data":"820c3557ddf91f67c781ddc9210d094a19c251f3d3001747d32ca3253aaf22ed"}
Jul 28 16:15:08.095870 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.095830    2575 generic.go:356] "Generic (PLEG): container finished" podID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerID="524ce9829651df63f20dc4b1ab9fec00bb947cc525568a12b5ace56426be07b1" exitCode=0
Jul 28 16:15:08.096255 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.095901    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" event={"ID":"575ece81-6bb8-4086-813f-d28b7c4eab37","Type":"ContainerDied","Data":"524ce9829651df63f20dc4b1ab9fec00bb947cc525568a12b5ace56426be07b1"}
Jul 28 16:15:08.190968 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.190942    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:15:08.277515 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.277421    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls\") pod \"575ece81-6bb8-4086-813f-d28b7c4eab37\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") "
Jul 28 16:15:08.277515 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.277468    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-rbac-proxy-sar-config\") pod \"575ece81-6bb8-4086-813f-d28b7c4eab37\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") "
Jul 28 16:15:08.277515 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.277490    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvcxn\" (UniqueName: \"kubernetes.io/projected/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-api-access-jvcxn\") pod \"575ece81-6bb8-4086-813f-d28b7c4eab37\" (UID: \"575ece81-6bb8-4086-813f-d28b7c4eab37\") "
Jul 28 16:15:08.277855 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.277831    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "575ece81-6bb8-4086-813f-d28b7c4eab37" (UID: "575ece81-6bb8-4086-813f-d28b7c4eab37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:08.279690 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.279659    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "575ece81-6bb8-4086-813f-d28b7c4eab37" (UID: "575ece81-6bb8-4086-813f-d28b7c4eab37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:15:08.279812 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.279687    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-api-access-jvcxn" (OuterVolumeSpecName: "kube-api-access-jvcxn") pod "575ece81-6bb8-4086-813f-d28b7c4eab37" (UID: "575ece81-6bb8-4086-813f-d28b7c4eab37"). InnerVolumeSpecName "kube-api-access-jvcxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:15:08.378183 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.378144    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/575ece81-6bb8-4086-813f-d28b7c4eab37-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:15:08.378183 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.378187    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:15:08.378393 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:08.378198    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvcxn\" (UniqueName: \"kubernetes.io/projected/575ece81-6bb8-4086-813f-d28b7c4eab37-kube-api-access-jvcxn\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:15:09.102255 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:09.102220    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7" event={"ID":"575ece81-6bb8-4086-813f-d28b7c4eab37","Type":"ContainerDied","Data":"966220e09a2060b63e060bebef201dcad6fec306fb7260f6d1897c53ca858f0c"}
Jul 28 16:15:09.102740 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:09.102293    2575 scope.go:117] "RemoveContainer" containerID="820c3557ddf91f67c781ddc9210d094a19c251f3d3001747d32ca3253aaf22ed"
Jul 28 16:15:09.102740 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:09.102300    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"
Jul 28 16:15:09.111649 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:09.111632    2575 scope.go:117] "RemoveContainer" containerID="524ce9829651df63f20dc4b1ab9fec00bb947cc525568a12b5ace56426be07b1"
Jul 28 16:15:09.128744 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:09.128719    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"]
Jul 28 16:15:09.136549 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:09.136521    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd6ec-predictor-6948f85895-wp4q7"]
Jul 28 16:15:10.257244 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:15:10.257212    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" path="/var/lib/kubelet/pods/575ece81-6bb8-4086-813f-d28b7c4eab37/volumes"
Jul 28 16:17:44.379993 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:17:44.379898    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:17:44.384605 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:17:44.384583    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:17:44.385979 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:17:44.385959    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:17:44.390801 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:17:44.390785    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:22:29.420020 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:29.419943    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"]
Jul 28 16:22:29.420552 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:29.420214    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container" containerID="cri-o://c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051" gracePeriod=30
Jul 28 16:22:29.420552 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:29.420287    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kube-rbac-proxy" containerID="cri-o://ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d" gracePeriod=30
Jul 28 16:22:30.126328 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:30.126285    2575 generic.go:356] "Generic (PLEG): container finished" podID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerID="ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d" exitCode=2
Jul 28 16:22:30.126504 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:30.126347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" event={"ID":"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd","Type":"ContainerDied","Data":"ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d"}
Jul 28 16:22:32.467117 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.467087    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:22:32.549592 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.549504    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls\") pod \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") "
Jul 28 16:22:32.549592 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.549574    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-rbac-proxy-sar-config\") pod \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") "
Jul 28 16:22:32.549592 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.549595    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk7g9\" (UniqueName: \"kubernetes.io/projected/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-api-access-lk7g9\") pod \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\" (UID: \"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd\") "
Jul 28 16:22:32.549950 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.549927    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" (UID: "bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:32.551748 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.551727    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" (UID: "bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:32.551748 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.551740    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-api-access-lk7g9" (OuterVolumeSpecName: "kube-api-access-lk7g9") pod "bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" (UID: "bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd"). InnerVolumeSpecName "kube-api-access-lk7g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:32.650620 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.650583    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-proxy-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:22:32.650620 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.650615    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:22:32.650620 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:32.650627    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7g9\" (UniqueName: \"kubernetes.io/projected/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd-kube-api-access-lk7g9\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\""
Jul 28 16:22:33.142696 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.142653    2575 generic.go:356] "Generic (PLEG): container finished" podID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerID="c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051" exitCode=0
Jul 28 16:22:33.142696 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.142689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" event={"ID":"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd","Type":"ContainerDied","Data":"c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051"}
Jul 28 16:22:33.142923 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.142710    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"
Jul 28 16:22:33.142923 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.142761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf" event={"ID":"bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd","Type":"ContainerDied","Data":"144cafa1e65f3c915a08c95ba44a62498cb345c2bceaef084cf342107852a283"}
Jul 28 16:22:33.142923 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.142792    2575 scope.go:117] "RemoveContainer" containerID="ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d"
Jul 28 16:22:33.151567 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.151552    2575 scope.go:117] "RemoveContainer" containerID="c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051"
Jul 28 16:22:33.158928 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.158911    2575 scope.go:117] "RemoveContainer" containerID="ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d"
Jul 28 16:22:33.159204 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:22:33.159180    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d\": container with ID starting with ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d not found: ID does not exist" containerID="ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d"
Jul 28 16:22:33.159270 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.159213    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d"} err="failed to get container status \"ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d\": rpc error: code = NotFound desc = could not find container \"ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d\": container with ID starting with ffaefb42a0cf1a4aa2e1059efd871b2c8865d7f703a12c87dc20a96d8b1fa57d not found: ID does not exist"
Jul 28 16:22:33.159270 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.159236    2575 scope.go:117] "RemoveContainer" containerID="c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051"
Jul 28 16:22:33.159465 ip-10-0-143-228 kubenswrapper[2575]: E0728 16:22:33.159448    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051\": container with ID starting with c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051 not found: ID does not exist" containerID="c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051"
Jul 28 16:22:33.159512 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.159472    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051"} err="failed to get container status \"c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051\": rpc error: code = NotFound desc = could not find container \"c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051\": container with ID starting with c3eaae2200c2abd9b3b43f350ad5020d715e095c07d40cec657ee264930ba051 not found: ID does not exist"
Jul 28 16:22:33.167274 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.167252    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"]
Jul 28 16:22:33.171179 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:33.171149    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd785-predictor-989fd784c-ndhgf"]
Jul 28 16:22:34.256488 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:34.256458    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" path="/var/lib/kubelet/pods/bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd/volumes"
Jul 28 16:22:44.408765 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:44.408662    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:22:44.425529 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:44.414154    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:22:44.425529 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:44.415081    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:22:44.425529 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:22:44.420623    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:23:05.104492 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:05.104456    2575 ???:1] "http: TLS handshake error from 10.0.134.118:45864: EOF"
Jul 28 16:23:05.109846 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:05.109820    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f5ms2_c75abded-e781-47c2-a233-e50561257d20/global-pull-secret-syncer/0.log"
Jul 28 16:23:05.275456 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:05.275421    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s7dpj_d0d40f3e-c36b-4def-acaa-dcf458c09666/konnectivity-agent/0.log"
Jul 28 16:23:05.354989 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:05.354912    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-228.ec2.internal_6807966616aea208bee6841100daf208/haproxy/0.log"
Jul 28 16:23:09.786378 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:09.786344    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38fb86fa-e411-45da-ba8c-43c74d956a19/alertmanager/0.log"
Jul 28 16:23:09.841872 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:09.841843    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38fb86fa-e411-45da-ba8c-43c74d956a19/config-reloader/0.log"
Jul 28 16:23:09.896725 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:09.896695    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38fb86fa-e411-45da-ba8c-43c74d956a19/kube-rbac-proxy-web/0.log"
Jul 28 16:23:09.940268 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:09.940240    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38fb86fa-e411-45da-ba8c-43c74d956a19/kube-rbac-proxy/0.log"
Jul 28 16:23:09.982846 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:09.982811    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38fb86fa-e411-45da-ba8c-43c74d956a19/kube-rbac-proxy-metric/0.log"
Jul 28 16:23:10.031718 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:10.031693    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38fb86fa-e411-45da-ba8c-43c74d956a19/prom-label-proxy/0.log"
Jul 28 16:23:10.125715 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:10.125637    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38fb86fa-e411-45da-ba8c-43c74d956a19/init-config-reloader/0.log"
Jul 28 16:23:10.845499 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:10.845467    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vspq2_facb0c31-45fd-47c4-8d9f-b9b86b8bcc62/node-exporter/0.log"
Jul 28 16:23:10.900673 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:10.900632    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vspq2_facb0c31-45fd-47c4-8d9f-b9b86b8bcc62/kube-rbac-proxy/0.log"
Jul 28 16:23:10.941449 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:10.941422    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vspq2_facb0c31-45fd-47c4-8d9f-b9b86b8bcc62/init-textfile/0.log"
Jul 28 16:23:11.758745 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:11.758712    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-dqcch_7b139057-3b86-4cd8-a749-c9185310e0dc/prometheus-operator/0.log"
Jul 28 16:23:11.863468 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:11.863435    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-dqcch_7b139057-3b86-4cd8-a749-c9185310e0dc/kube-rbac-proxy/0.log"
Jul 28 16:23:11.946425 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:11.946396    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-pv5nh_ea384ab4-a0d6-4275-bb56-e2017933fdd0/prometheus-operator-admission-webhook/0.log"
Jul 28 16:23:12.060626 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.060546    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bb8659964-szppw_843c4461-1c35-4d98-80b6-4c80ec996f95/telemeter-client/0.log"
Jul 28 16:23:12.124392 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.124364    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bb8659964-szppw_843c4461-1c35-4d98-80b6-4c80ec996f95/reload/0.log"
Jul 28 16:23:12.171298 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.171269    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bb8659964-szppw_843c4461-1c35-4d98-80b6-4c80ec996f95/kube-rbac-proxy/0.log"
Jul 28 16:23:12.234347 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.234320    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558bc6f974-5kv4r_fbaa0008-e48f-4b2f-9170-1d2a9af61125/thanos-query/0.log"
Jul 28 16:23:12.290282 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.290253    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558bc6f974-5kv4r_fbaa0008-e48f-4b2f-9170-1d2a9af61125/kube-rbac-proxy-web/0.log"
Jul 28 16:23:12.341219 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.341136    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558bc6f974-5kv4r_fbaa0008-e48f-4b2f-9170-1d2a9af61125/kube-rbac-proxy/0.log"
Jul 28 16:23:12.382741 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.382717    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558bc6f974-5kv4r_fbaa0008-e48f-4b2f-9170-1d2a9af61125/prom-label-proxy/0.log"
Jul 28 16:23:12.465984 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.465961    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558bc6f974-5kv4r_fbaa0008-e48f-4b2f-9170-1d2a9af61125/kube-rbac-proxy-rules/0.log"
Jul 28 16:23:12.523013 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.522990    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558bc6f974-5kv4r_fbaa0008-e48f-4b2f-9170-1d2a9af61125/kube-rbac-proxy-metrics/0.log"
Jul 28 16:23:12.916286 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916253    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"]
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916585    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kube-rbac-proxy"
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916596    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kube-rbac-proxy"
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916617    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container"
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916623    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container"
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916635    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container"
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916641    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container"
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916650    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kube-rbac-proxy"
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916657    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kube-rbac-proxy"
Jul 28 16:23:12.916659 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916663    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916669    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916676    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kube-rbac-proxy"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916681    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kube-rbac-proxy"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916731    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kube-rbac-proxy"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916738    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kube-rbac-proxy"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916744    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce3f050-9c6b-4a4f-a217-cbc4ab0fd4dd" containerName="kserve-container"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916752    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kserve-container"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916759    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4bb668c-0cfe-49e7-9ddd-1c37b7458369" containerName="kserve-container"
Jul 28 16:23:12.916939 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.916766    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="575ece81-6bb8-4086-813f-d28b7c4eab37" containerName="kube-rbac-proxy"
Jul 28 16:23:12.919593 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.919577    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:12.925098 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.925077    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bns78\"/\"default-dockercfg-rvrtl\""
Jul 28 16:23:12.925098 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.925093    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bns78\"/\"openshift-service-ca.crt\""
Jul 28 16:23:12.926341 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.926322    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bns78\"/\"kube-root-ca.crt\""
Jul 28 16:23:12.934223 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.934204    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"]
Jul 28 16:23:12.997097 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.997062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dbq\" (UniqueName: \"kubernetes.io/projected/d4c33147-b48c-4e44-9bb9-a7f161af8f77-kube-api-access-v2dbq\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:12.997097 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.997109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-sys\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:12.997385 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.997214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-podres\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:12.997385 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.997245    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-proc\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:12.997385 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:12.997285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-lib-modules\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098633 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-lib-modules\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098789 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dbq\" (UniqueName: \"kubernetes.io/projected/d4c33147-b48c-4e44-9bb9-a7f161af8f77-kube-api-access-v2dbq\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098789 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098685    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-sys\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098789 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098718    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-podres\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098789 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-proc\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098963 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-lib-modules\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098963 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098802    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-sys\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098963 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-proc\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.098963 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.098850    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4c33147-b48c-4e44-9bb9-a7f161af8f77-podres\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.113184 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.113135    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dbq\" (UniqueName: \"kubernetes.io/projected/d4c33147-b48c-4e44-9bb9-a7f161af8f77-kube-api-access-v2dbq\") pod \"perf-node-gather-daemonset-fmgb4\" (UID: \"d4c33147-b48c-4e44-9bb9-a7f161af8f77\") " pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.228956 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.228926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:13.368959 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.368907    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"]
Jul 28 16:23:13.373639 ip-10-0-143-228 kubenswrapper[2575]: W0728 16:23:13.373611    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4c33147_b48c_4e44_9bb9_a7f161af8f77.slice/crio-c255a52f26250fd0a8fbbbf83ccada93c0262559b6f9f2be9d6841e1abfe8467 WatchSource:0}: Error finding container c255a52f26250fd0a8fbbbf83ccada93c0262559b6f9f2be9d6841e1abfe8467: Status 404 returned error can't find the container with id c255a52f26250fd0a8fbbbf83ccada93c0262559b6f9f2be9d6841e1abfe8467
Jul 28 16:23:13.666884 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:13.666853    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-b9tl6_988272ab-9de5-4cce-a773-c91dc24e3a3f/networking-console-plugin/0.log"
Jul 28 16:23:14.314833 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:14.314809    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/2.log"
Jul 28 16:23:14.319142 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:14.319116    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ps655_687b4a32-0489-4bb5-9e72-47bab2e0702e/console-operator/3.log"
Jul 28 16:23:14.329214 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:14.329188    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4" event={"ID":"d4c33147-b48c-4e44-9bb9-a7f161af8f77","Type":"ContainerStarted","Data":"a4a1461bc0681443f1ef2158f19265de7fbae72334bd676c38a527e6a6fec9bd"}
Jul 28 16:23:14.329323 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:14.329248    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4" event={"ID":"d4c33147-b48c-4e44-9bb9-a7f161af8f77","Type":"ContainerStarted","Data":"c255a52f26250fd0a8fbbbf83ccada93c0262559b6f9f2be9d6841e1abfe8467"}
Jul 28 16:23:14.329323 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:14.329280    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:14.352426 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:14.352385    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4" podStartSLOduration=2.352371029 podStartE2EDuration="2.352371029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:23:14.351480628 +0000 UTC m=+3030.601537626" watchObservedRunningTime="2026-07-28 16:23:14.352371029 +0000 UTC m=+3030.602428033"
Jul 28 16:23:15.373593 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:15.373561    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-r6wcc_34e5b2fb-cd79-42ea-9e4d-ae5617cf20d6/volume-data-source-validator/0.log"
Jul 28 16:23:16.281292 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:16.281262    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mznsm_9d0c8519-a071-4afa-a5ab-5bc282690423/dns/0.log"
Jul 28 16:23:16.305873 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:16.305850    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mznsm_9d0c8519-a071-4afa-a5ab-5bc282690423/kube-rbac-proxy/0.log"
Jul 28 16:23:16.445748 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:16.445716    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rr2gh_30a8b0f0-8c63-4b32-a1e5-8c2b4bcd9882/dns-node-resolver/0.log"
Jul 28 16:23:16.948761 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:16.948733    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8v5vg_5c0a5568-31d1-4001-b029-47265e985078/node-ca/0.log"
Jul 28 16:23:18.324385 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:18.324353    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sqzxh_c4c0ca25-e742-4a8c-b5cd-1e9f36cbea7a/serve-healthcheck-canary/0.log"
Jul 28 16:23:18.763778 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:18.763752    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-d229l_1f021571-3315-4bf9-889b-110d52285a46/insights-operator/0.log"
Jul 28 16:23:18.764193 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:18.764156    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-d229l_1f021571-3315-4bf9-889b-110d52285a46/insights-operator/1.log"
Jul 28 16:23:18.869636 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:18.869607    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k99wm_f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d/kube-rbac-proxy/0.log"
Jul 28 16:23:18.891708 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:18.891687    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k99wm_f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d/exporter/0.log"
Jul 28 16:23:18.916026 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:18.915993    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k99wm_f3a0a4ce-da75-4e4e-a659-ff20f6f7dc1d/extractor/0.log"
Jul 28 16:23:20.343711 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:20.343685    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bns78/perf-node-gather-daemonset-fmgb4"
Jul 28 16:23:21.423260 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:21.423229    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-bghcp_1d0f2487-d84b-434d-bdd6-d610bbb1b218/server/0.log"
Jul 28 16:23:21.595419 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:21.595383    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dvwl4_78071dbf-7ee4-41dd-b698-1402f7552a9a/s3-init/0.log"
Jul 28 16:23:21.619252 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:21.619219    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dvwl4_78071dbf-7ee4-41dd-b698-1402f7552a9a/download-hf-model/0.log"
Jul 28 16:23:21.674439 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:21.674415    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-zs2cf_44837f84-8631-488c-ab11-6b94d1bb0b87/seaweedfs/0.log"
Jul 28 16:23:27.889752 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:27.889725    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9cqsh_f571f6f2-c1d0-4ef1-a0b4-3df8e717b9b1/kube-multus/0.log"
Jul 28 16:23:27.926336 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:27.926310    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zfcm_38e5b545-9292-4266-bdc0-63ba5e9c70e3/kube-multus-additional-cni-plugins/0.log"
Jul 28 16:23:27.949821 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:27.949792    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zfcm_38e5b545-9292-4266-bdc0-63ba5e9c70e3/egress-router-binary-copy/0.log"
Jul 28 16:23:27.977152 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:27.977130    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zfcm_38e5b545-9292-4266-bdc0-63ba5e9c70e3/cni-plugins/0.log"
Jul 28 16:23:28.005634 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:28.005607    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zfcm_38e5b545-9292-4266-bdc0-63ba5e9c70e3/bond-cni-plugin/0.log"
Jul 28 16:23:28.034912 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:28.034892    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zfcm_38e5b545-9292-4266-bdc0-63ba5e9c70e3/routeoverride-cni/0.log"
Jul 28 16:23:28.060305 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:28.060276    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zfcm_38e5b545-9292-4266-bdc0-63ba5e9c70e3/whereabouts-cni-bincopy/0.log"
Jul 28 16:23:28.085524 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:28.085502    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zfcm_38e5b545-9292-4266-bdc0-63ba5e9c70e3/whereabouts-cni/0.log"
Jul 28 16:23:28.721013 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:28.720937    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lx8xf_9ab5b87b-81cc-4429-8147-c990db710694/network-metrics-daemon/0.log"
Jul 28 16:23:28.768644 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:28.768616    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lx8xf_9ab5b87b-81cc-4429-8147-c990db710694/kube-rbac-proxy/0.log"
Jul 28 16:23:29.790160 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:29.790130    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-controller/0.log"
Jul 28 16:23:29.819844 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:29.819820    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/1.log"
Jul 28 16:23:29.820359 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:29.820340    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/ovn-acl-logging/0.log"
Jul 28 16:23:29.843924 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:29.843903    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/kube-rbac-proxy-node/0.log"
Jul 28 16:23:29.872415 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:29.872393    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 16:23:29.896036 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:29.896008    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/northd/0.log"
Jul 28 16:23:29.918589 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:29.918564    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/nbdb/0.log"
Jul 28 16:23:29.961376 ip-10-0-143-228 kubenswrapper[2575]: I0728 16:23:29.961340    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fk7vh_74285376-f5c6-4e10-a103-9fd61945f750/sbdb/0.log"