Jul 08 16:34:16.994534 ip-10-0-135-143 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.470079    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474467    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474478    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474483    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474486    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474489    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474492    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474495    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:17.506100 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474498    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474501    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474503    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474506    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474508    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474511    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474513    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474516    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474522    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474525    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474528    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474530    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474533    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474536    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474539    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474541    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474544    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474546    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474550    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:17.513286 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474552    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474555    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474560    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474564    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474568    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474571    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474573    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474576    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474579    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474582    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474584    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474586    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474589    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474591    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474594    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474596    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474599    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474601    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474603    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474606    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:17.514154 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474609    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474611    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474614    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474617    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474619    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474622    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474625    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474628    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474631    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474633    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474636    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474639    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474641    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474644    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474647    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474649    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474654    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474656    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474659    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474661    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:17.515012 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474664    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474666    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474668    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474672    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474675    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474678    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474680    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474682    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474685    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474688    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474690    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474693    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474695    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474698    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474700    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474705    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474708    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474711    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474714    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474716    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:17.515902 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474719    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474722    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474725    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474728    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474730    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474733    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474735    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474738    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474740    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474743    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474745    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474748    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.474752    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478328    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478338    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478343    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478347    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478351    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478354    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:17.516593 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478357    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478360    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478362    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478366    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478369    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478371    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478374    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478378    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478382    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478384    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478387    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478390    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478392    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478395    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478398    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478401    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478404    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478406    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478409    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478412    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:17.517134 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478414    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478417    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478420    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478423    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478428    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478431    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478434    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478436    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478439    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478442    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478445    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478447    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478449    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478452    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478454    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478457    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478459    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478462    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478464    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:17.517711 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478467    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478469    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478472    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478474    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478477    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478480    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478483    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478486    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478488    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478491    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478493    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478496    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478499    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478501    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478503    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478506    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478508    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478512    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478514    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478516    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:17.518250 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478519    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478521    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478524    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478529    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478531    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478533    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478536    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478538    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478541    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478543    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478546    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478548    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478551    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478553    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478556    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478559    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478562    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478565    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478567    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478570    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:17.518778 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478572    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478575    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478577    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478580    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478582    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478584    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478587    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478589    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478591    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478594    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478596    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478599    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478601    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478604    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478955    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478961    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478964    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478967    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478970    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478972    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:17.519351 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478975    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478977    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478980    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478982    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478985    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478987    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478990    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478993    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478995    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.478998    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479001    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479005    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479009    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479012    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479015    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479017    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479020    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479022    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479025    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479027    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:17.519887 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479030    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479032    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479035    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479038    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479040    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479043    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479045    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479049    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479052    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479055    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479058    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479060    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479063    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479065    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479068    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479070    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479073    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479076    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479078    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479081    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:17.610357 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479083    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:17.539798 ip-10-0-135-143 systemd[1]: Started Kubernetes Kubelet.
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479085    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479088    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479091    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479093    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479096    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479100    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479104    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479107    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479110    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479112    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479115    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479117    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479120    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479122    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479125    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479127    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479130    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479132    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:17.611206 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479135    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479138    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479141    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479143    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479145    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479148    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479151    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479153    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479156    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479159    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479161    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479164    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479166    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479169    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479172    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479174    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479177    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479181    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479184    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:17.611972 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479188    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479191    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479193    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479196    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479199    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479201    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479203    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479206    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479208    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479211    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479213    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479216    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479218    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479222    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479225    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479307    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479314    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479320    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479324    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479328    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479331    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 16:34:17.612598 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479335    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479339    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479342    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479345    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479348    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479351    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479357    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479360    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479362    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479365    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479369    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479372    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479376    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479379    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479382    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479385    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479388    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479391    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479394    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479397    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479400    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479404    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479407    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479409    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 16:34:17.613280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479414    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479417    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479420    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479423    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479426    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479429    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479432    2575 flags.go:64] FLAG: --enable-server="true"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479435    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479439    2575 flags.go:64] FLAG: --event-burst="100"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479442    2575 flags.go:64] FLAG: --event-qps="50"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479444    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479447    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479450    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479453    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479458    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479460    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479463    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479467    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479469    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479473    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479476    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479478    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479481    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479484    2575 flags.go:64] FLAG: --feature-gates=""
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479488    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 16:34:17.614017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479490    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479493    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479497    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479500    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479503    2575 flags.go:64] FLAG: --help="false"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479505    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-143.ec2.internal"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479508    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479514    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479516    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479520    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479523    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479526    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479529    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479531    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479534    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479537    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479540    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479543    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479546    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479549    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479551    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479555    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479558    2575 flags.go:64] FLAG: --lock-file=""
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479561    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 16:34:17.614832 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479564    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479567    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479572    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479575    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479578    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479581    2575 flags.go:64] FLAG: --logging-format="text"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479584    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479587    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479589    2575 flags.go:64] FLAG: --manifest-url=""
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479592    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479596    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479599    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479603    2575 flags.go:64] FLAG: --max-pods="110"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479606    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479609    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479613    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479616    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479619    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479622    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479625    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479632    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479635    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479638    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 16:34:17.615573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479641    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479644    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479649    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479652    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479655    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479658    2575 flags.go:64] FLAG: --port="10250"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479663    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479666    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-046e6af718195e898"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479669    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479671    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479674    2575 flags.go:64] FLAG: --register-node="true"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479677    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479681    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479685    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479687    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479690    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479694    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479697    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479700    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479703    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479706    2575 flags.go:64] FLAG: --runonce="false"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479708    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479711    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479714    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479717    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 16:34:17.616305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479721    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479724    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479727    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479730    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479733    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479736    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479739    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479742    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479745    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479748    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479750    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479755    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479758    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479763    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479767    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479769    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479772    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479775    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479778    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479781    2575 flags.go:64] FLAG: --v="2"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479784    2575 flags.go:64] FLAG: --version="false"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479789    2575 flags.go:64] FLAG: --vmodule=""
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479792    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.479795    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 16:34:17.617375 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479902    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479907    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479910    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479913    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479917    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479919    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479922    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479925    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479929    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479932    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479935    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479937    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479941    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479943    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479946    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479949    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479951    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479954    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479956    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479958    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:17.618127 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479961    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479965    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479968    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479970    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479973    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479976    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479978    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479981    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479983    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479986    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479989    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479992    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479994    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479996    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.479999    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480001    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480004    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480006    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480009    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480011    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:19.609099 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480015    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480018    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480020    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480023    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480026    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480028    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480031    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480033    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480036    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480038    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480041    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480044    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480046    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480051    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480053    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480055    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480058    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480060    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480063    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480066    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:19.610117 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480068    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480070    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480073    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480076    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480078    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480081    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480083    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480086    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480088    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480090    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480093    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480097    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480101    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480104    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480107    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480110    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480113    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480116    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480118    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:19.610901 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480121    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480125    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480128    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480131    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480134    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480136    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480140    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480143    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480145    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480147    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480150    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480153    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480156    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480158    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480161    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480163    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480166    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480169    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480171    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:19.611499 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.480174    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.480182    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.486612    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.486626    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.486643    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.486648    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486704    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486709    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486712    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486719    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486722    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486725    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486728    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486731    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486734    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:19.612115 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486737    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486739    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486742    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486744    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486747    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486749    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486752    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486754    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486758    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486761    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486763    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486766    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486769    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486771    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486774    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486777    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486780    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486784    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486787    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:19.612915 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486791    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486793    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486812    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486816    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486819    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486822    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486825    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486830    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486832    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486835    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486838    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486840    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486843    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486846    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486848    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486851    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486854    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486856    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486859    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486861    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:19.613431 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486864    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486868    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486870    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486872    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486875    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486878    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486880    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486883    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486885    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486888    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486891    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486894    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486897    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486899    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486902    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486904    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486907    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486909    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486911    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486915    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:19.614028 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486918    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486920    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486923    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486926    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486929    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486931    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486934    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486937    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486939    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486942    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486944    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486948    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486952    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486956    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486958    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486961    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486963    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486966    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486968    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486971    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:19.614596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486973    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486976    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486979    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486982    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486984    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486987    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486990    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486992    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486994    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.486997    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487001    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.487007    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487113    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487117    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:19.615207 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487120    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487122    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487125    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487128    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487131    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487134    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487136    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487139    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487142    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487145    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487148    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487151    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487154    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487157    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487159    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487162    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487164    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487167    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487170    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:19.615602 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487172    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487175    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487178    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487180    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487183    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487185    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487188    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487190    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487192    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487195    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487197    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487202    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487204    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487207    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487212    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487215    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487217    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487220    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487223    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:19.616161 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487226    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487230    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487233    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487235    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487238    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487243    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487245    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487248    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487251    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487253    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487256    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487258    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487260    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487278    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487283    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487286    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487290    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487292    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487295    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:19.616677 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487297    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487300    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487302    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487305    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487307    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487311    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487313    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487316    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487319    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487321    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487323    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487326    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487328    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487331    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487333    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487336    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487338    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487341    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487345    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487347    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:19.617230 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487350    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487352    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487354    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487357    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487360    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487362    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487364    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487366    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487369    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487371    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487375    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487377    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487379    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487382    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487384    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487387    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487389    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487393    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487396    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:19.617785 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:17.487398    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.487403    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.487489    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.490345    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.491740    2575 server.go:1019] "Starting client certificate rotation"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.491840    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.491903    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.494293    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.495287    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.504434    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.506380    2575 log.go:25] "Validated CRI v1 image API"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.507251    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 16:34:19.618332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.514156    2575 fs.go:135] Filesystem UUIDs: map[62a4eca6-8576-4ee6-b6f8-5f6e140c5012:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 be0321e4-0869-4d23-92f5-b7c78b8ef1c9:/dev/nvme0n1p4]
Jul 08 16:34:19.618692 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.514173    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 16:34:19.618798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.519294    2575 manager.go:216] Machine: {Timestamp:2026-07-08 16:34:17.518103393 +0000 UTC m=+0.364040712 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097662 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ffced540af31fe9c62ba3e7165e2d SystemUUID:ec2ffced-540a-f31f-e9c6-2ba3e7165e2d BootID:5ad2270f-c3ce-44ad-b6dc-8599ab8da41f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a0:de:40:31:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a0:de:40:31:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:7f:61:62:ee:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 16:34:19.618798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.519394    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 16:34:19.618798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.519483    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 16:34:19.618798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.522580    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 16:34:19.618798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.522605    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 16:34:19.618798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.523481    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 16:34:19.618798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.523492    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 16:34:19.618798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.523500    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.523512    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.524885    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.527758    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.527880    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.528658    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.528673    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.528686    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.528695    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.528705    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.528978    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.528993    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.531543    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.533554    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.533594    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.533638    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534872    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534891    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534900    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534908    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534920    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534932    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534942    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534958    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534965    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534971    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534985    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.534990    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.536286    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.536294    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.539073    2575 server.go:1268] "Started kubelet"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.539138    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.539144    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.539205    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.545848    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.548807    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.548811    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.549478    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.549563    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 16:34:19.619070 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.549688    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.552959    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.553345    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.553889    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.553928    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.554366    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.554381    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.554602    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.554613    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.555181    2575 factory.go:55] Registering systemd factory
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.555196    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.555395    2575 factory.go:153] Registering CRI-O factory
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.555411    2575 factory.go:223] Registration of the crio container factory successfully
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.555461    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.555486    2575 factory.go:103] Registering Raw factory
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.555501    2575 manager.go:1196] Started watching for new ooms in manager
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.555873    2575 manager.go:349] Starting recovery of all containers
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.560383    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.560483    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.560248    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18c05daebe6a542b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-08 16:34:17.539048491 +0000 UTC m=+0.384985787,LastTimestamp:2026-07-08 16:34:17.539048491 +0000 UTC m=+0.384985787,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.566524    2575 manager.go:354] Recovery completed
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.571752    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.573369    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.573406    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.573419    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.574355    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.574364    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.574378    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.574409    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.575084    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.575119    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.575135    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.575685    2575 policy_none.go:49] "None policy: Start"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.575698    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 16:34:19.620212 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.575711    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.575871    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18c05daec0765f5f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-08 16:34:17.573392223 +0000 UTC m=+0.419329520,LastTimestamp:2026-07-08 16:34:17.573392223 +0000 UTC m=+0.419329520,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.576764    2575 policy_none.go:47] "Start"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.585733    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18c05daec076ac7b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-08 16:34:17.573411963 +0000 UTC m=+0.419349261,LastTimestamp:2026-07-08 16:34:17.573411963 +0000 UTC m=+0.419349261,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.597953    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18c05daec076e129  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-08 16:34:17.573425449 +0000 UTC m=+0.419362748,LastTimestamp:2026-07-08 16:34:17.573425449 +0000 UTC m=+0.419362748,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.604809    2575 manager.go:337] "Starting Device Plugin manager"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.604833    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.604841    2575 server.go:87] "Starting device plugin registration server"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.605096    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.605107    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.605163    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.605235    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.605242    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.605806    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.605853    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:19.621405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.608078    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvtj8"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:17.611671    2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-143.ec2.internal.18c05daec0765f5f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18c05daec0765f5f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-08 16:34:17.573392223 +0000 UTC m=+0.419329520,LastTimestamp:2026-07-08 16:34:17.575094404 +0000 UTC m=+0.421031701,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:17.615954    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvtj8"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.562830    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.564047    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.564064    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.564074    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.564095    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.572961    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.572980    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-143.ec2.internal\": node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.587950    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.601264    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.602456    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.602491    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.602518    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.602556    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 16:34:19.621909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.604144    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.688259    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.703394    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"]
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.703445    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.704256    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.704275    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.704283    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.705505    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.705634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.705678    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.706163    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.706189    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.706199    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.706374    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.706384    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.706391    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.707633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.707668    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.708405    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.708426    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.708435    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.713889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c73243dc8a0860bf821a5ab616d0670d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"c73243dc8a0860bf821a5ab616d0670d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.713914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f3918137afdb7026e67d79021b1461d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"3f3918137afdb7026e67d79021b1461d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.713932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f3918137afdb7026e67d79021b1461d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"3f3918137afdb7026e67d79021b1461d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.727202    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.754904 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.731106    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.895684 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.788498    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:19.895684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.814163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c73243dc8a0860bf821a5ab616d0670d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"c73243dc8a0860bf821a5ab616d0670d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.895684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.814192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f3918137afdb7026e67d79021b1461d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"3f3918137afdb7026e67d79021b1461d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.895684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.814208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f3918137afdb7026e67d79021b1461d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"3f3918137afdb7026e67d79021b1461d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.895684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.814239    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f3918137afdb7026e67d79021b1461d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"3f3918137afdb7026e67d79021b1461d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.895684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.814251    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c73243dc8a0860bf821a5ab616d0670d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"c73243dc8a0860bf821a5ab616d0670d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.895684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:19.814265    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f3918137afdb7026e67d79021b1461d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"3f3918137afdb7026e67d79021b1461d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:19.895684 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.889563    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.035121 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:19.989960    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.035121 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.031441    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:20.035896 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.035127    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:20.186020 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:20.090161    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.363313 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:20.190596    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.363313 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:20.291028    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.363313 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.347472    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 16:34:20.363313 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.347567    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 16:34:20.486096 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:20.391948    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.486096 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.442416    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:20.486096 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.444258    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:20.486096 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.444279    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:20.486096 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.444288    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:20.486096 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.474447    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 16:29:19 +0000 UTC" deadline="2028-01-18 00:18:31.061317467 +0000 UTC"
Jul 08 16:34:20.486096 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.474468    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13399h44m10.586851963s"
Jul 08 16:34:20.493341 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:20.492584    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.624606 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:20.593057    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.624606 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.607172    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:20.806232 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:20.630640    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3918137afdb7026e67d79021b1461d.slice/crio-1a5221eb8fe2c804556bab7cb7ec53a7d94915b0252330c82451ee2a1ce8ca91 WatchSource:0}: Error finding container 1a5221eb8fe2c804556bab7cb7ec53a7d94915b0252330c82451ee2a1ce8ca91: Status 404 returned error can't find the container with id 1a5221eb8fe2c804556bab7cb7ec53a7d94915b0252330c82451ee2a1ce8ca91
Jul 08 16:34:20.806232 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.637544    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:34:20.806232 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:20.687519    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73243dc8a0860bf821a5ab616d0670d.slice/crio-8c0f3e525b0dbee445089c18b876e8d67872567828241afe94562e54ad40bc3d WatchSource:0}: Error finding container 8c0f3e525b0dbee445089c18b876e8d67872567828241afe94562e54ad40bc3d: Status 404 returned error can't find the container with id 8c0f3e525b0dbee445089c18b876e8d67872567828241afe94562e54ad40bc3d
Jul 08 16:34:20.806232 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:20.693140    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:20.806232 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.709912    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:20.806232 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:20.793980    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 08 16:34:21.001981 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.872246    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:21.001981 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.910480    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:21.001981 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.924802    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:21.001981 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.925766    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 08 16:34:21.001981 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:20.939005    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:21.387582 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.387521    2575 apiserver.go:52] "Watching apiserver"
Jul 08 16:34:21.395565 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.395540    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 16:34:21.396630 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.396602    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-4x4wh","openshift-network-operator/iptables-alerter-gjvrt","openshift-ovn-kubernetes/ovnkube-node-8pcqm","kube-system/global-pull-secret-syncer-v8x8z","kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb","openshift-cluster-node-tuning-operator/tuned-vwl98","openshift-dns/node-resolver-vtcrd","openshift-image-registry/node-ca-8p82f","openshift-network-diagnostics/network-check-target-dtcdw","kube-system/konnectivity-agent-h8snv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal","openshift-multus/multus-6bqfr","openshift-multus/multus-additional-cni-plugins-bx962"]
Jul 08 16:34:21.398146 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.398124    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.399245 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.399222    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:21.400362 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.400340    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:21.402589 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.401569    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rz62j\""
Jul 08 16:34:21.402589 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.402090    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:21.402589 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.402194    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 16:34:21.403320 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.403045    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 16:34:21.403320 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.403156    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jrnql\""
Jul 08 16:34:21.405143 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.403620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.406349 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.406329    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.406447 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.406426    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:21.407150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.406829    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mg2dt\""
Jul 08 16:34:21.407150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.406669    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 16:34:21.407150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.407038    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 16:34:21.407150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.407114    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 16:34:21.407150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.407118    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 16:34:21.407420 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.407309    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 16:34:21.407860 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.407722    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 16:34:21.408077 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.408060    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.410953 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.410716    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 16:34:21.410953 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.410844    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gpjbp\""
Jul 08 16:34:21.410953 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.410904    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 16:34:21.411138 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.411113    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 16:34:21.411138 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.411118    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 16:34:21.411338 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.411321    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 16:34:21.411571 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.411529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.412062 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.412038    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:21.412807 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.412768    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.414119 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.414097    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.414796 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.414767    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 16:34:21.414874 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.414809    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 16:34:21.414943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.414896    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 16:34:21.415075 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.415050    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r9kqb\""
Jul 08 16:34:21.415411 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.415200    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 16:34:21.415411 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.415359    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 16:34:21.415411 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.415400    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wg2vj\""
Jul 08 16:34:21.415587 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.415553    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 16:34:21.415665 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.415634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.416107 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.416090    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 16:34:21.416271 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.416249    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lqszb\""
Jul 08 16:34:21.417188 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.417170    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:21.417277 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.417232    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:21.417943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.417752    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:21.417943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.417797    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 16:34:21.417943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.417797    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:21.418167 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.418052    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vvnmm\""
Jul 08 16:34:21.418965 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.418918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:21.419053 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.419027    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:21.420759 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.420740    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.423066 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.423046    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 16:34:21.423191 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.423172    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 16:34:21.423824 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.423806    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qztvz\""
Jul 08 16:34:21.425496 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.425475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysctl-d\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.425667 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.425635    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-slash\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.425792 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.425490    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:21.425858 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.425787    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-env-overrides\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.425910 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.425858    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-node-log\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.425959 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.425915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.426010 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.425978    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-systemd\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.426059 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426019    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-run\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.426106 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a975ef5-c4da-4258-a07d-41d5259defde-konnectivity-ca\") pod \"konnectivity-agent-h8snv\" (UID: \"1a975ef5-c4da-4258-a07d-41d5259defde\") " pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:21.426106 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.426164 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426144    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-sys-fs\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.426211 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426167    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-tuned\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.426211 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426196    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a975ef5-c4da-4258-a07d-41d5259defde-agent-certs\") pod \"konnectivity-agent-h8snv\" (UID: \"1a975ef5-c4da-4258-a07d-41d5259defde\") " pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:21.426307 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-socket-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.426307 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-modprobe-d\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.426307 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426292    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426443 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426323    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-ovn\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426443 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426345    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426443 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-cni-bin\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426443 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426641 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfrn\" (UniqueName: \"kubernetes.io/projected/2840b062-0201-4147-bc5e-17ceb2652dc7-kube-api-access-jrfrn\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.426641 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-sys\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.426641 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-log-socket\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426641 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.426641 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysconfig\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.426641 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-kubelet\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.426899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-run-netns\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-system-cni-dir\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.426899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-tmp\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.426899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovnkube-config\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.426899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-device-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.427129 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-os-release\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.427129 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hv56\" (UniqueName: \"kubernetes.io/projected/ae8b5950-ddff-413c-946f-20b797585a14-kube-api-access-7hv56\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.427129 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.426982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysctl-conf\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.427129 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-var-lib-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.427129 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427053    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovnkube-script-lib\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.427129 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427073    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-dbus\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.427129 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69srp\" (UniqueName: \"kubernetes.io/projected/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-kube-api-access-69srp\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.427129 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-systemd\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.427448 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovn-node-metrics-cert\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.427448 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427172    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-systemd-units\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.427448 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427187    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.427448 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.427448 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-registration-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.427448 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-kubernetes\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.427448 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-var-lib-kubelet\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.427448 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-host\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.427696 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-cni-netd\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.427696 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-kubelet-config\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.427696 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-etc-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.427839 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-lib-modules\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.427882 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktpg2\" (UniqueName: \"kubernetes.io/projected/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-kube-api-access-ktpg2\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.427958 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-cnibin\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.428004 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.427983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.453763 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.453738    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tdv5s"
Jul 08 16:34:21.462219 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.462196    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tdv5s"
Jul 08 16:34:21.512103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.512080    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 16:34:21.529070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529046    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-cni-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.529184 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-cni-multus\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.529184 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-kubernetes\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.529184 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-var-lib-kubelet\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.529345 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60e3631-e481-4969-9b02-2db258f40abc-host\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.529345 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529243    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-kubernetes\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.529345 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-system-cni-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.529345 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529298    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-var-lib-kubelet\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.529345 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-os-release\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529348    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-socket-dir-parent\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-conf-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsz77\" (UniqueName: \"kubernetes.io/projected/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-kube-api-access-vsz77\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-host\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-cni-netd\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-kubelet-config\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529497    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-host\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-daemon-config\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529527    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-cni-netd\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-etc-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-lib-modules\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktpg2\" (UniqueName: \"kubernetes.io/projected/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-kube-api-access-ktpg2\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.529581 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529589    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-etc-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-cnibin\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529667    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-kubelet-config\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6p5\" (UniqueName: \"kubernetes.io/projected/e60e3631-e481-4969-9b02-2db258f40abc-kube-api-access-9f6p5\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysctl-d\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529741    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-lib-modules\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-slash\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-env-overrides\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-node-log\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-cnibin\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-systemd\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-run\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.530144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529937    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-slash\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529714    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-cnibin\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530070    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysctl-d\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.529941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a975ef5-c4da-4258-a07d-41d5259defde-konnectivity-ca\") pod \"konnectivity-agent-h8snv\" (UID: \"1a975ef5-c4da-4258-a07d-41d5259defde\") " pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-cni-bin\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530163    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-node-log\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530165    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-multus-certs\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-sys-fs\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-tuned\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a975ef5-c4da-4258-a07d-41d5259defde-agent-certs\") pod \"konnectivity-agent-h8snv\" (UID: \"1a975ef5-c4da-4258-a07d-41d5259defde\") " pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.530279    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530307    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-socket-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530337    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:21.530843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.530349    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret podName:2a323b89-9740-4ab1-9d0d-1ca2bec3d93d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:22.030330156 +0000 UTC m=+3.019886898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret") pod "global-pull-secret-syncer-v8x8z" (UID: "2a323b89-9740-4ab1-9d0d-1ca2bec3d93d") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530360    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-modprobe-d\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530409    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-env-overrides\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530436    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-ovn\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530461    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-modprobe-d\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-cni-bin\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfrn\" (UniqueName: \"kubernetes.io/projected/2840b062-0201-4147-bc5e-17ceb2652dc7-kube-api-access-jrfrn\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-k8s-cni-cncf-io\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-sys\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.531405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530489    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-sys-fs\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-log-socket\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530701    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-etc-kubernetes\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370616ca-420a-4679-91d6-9c035007c735-tmp-dir\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530758    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-ovn\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysconfig\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530806    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530807    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530828    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysconfig\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-kubelet\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530870    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-cni-bin\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530881    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-kubelet\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.532060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530911    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzccj\" (UniqueName: \"kubernetes.io/projected/8751bafd-f26c-47bd-8fa8-f24755c1fb18-kube-api-access-kzccj\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530912    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a975ef5-c4da-4258-a07d-41d5259defde-konnectivity-ca\") pod \"konnectivity-agent-h8snv\" (UID: \"1a975ef5-c4da-4258-a07d-41d5259defde\") " pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-cni-binary-copy\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-run-netns\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530963    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-systemd\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.530982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-system-cni-dir\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-hostroot\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-tmp\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531073    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-run\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovnkube-config\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-device-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-netns\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531141    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-kubelet\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.532750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531200    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-sys\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531217    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-system-cni-dir\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531237    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-host-run-netns\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531522    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-log-socket\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531722    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovnkube-config\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531755    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/370616ca-420a-4679-91d6-9c035007c735-hosts-file\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-os-release\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531810    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-device-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hv56\" (UniqueName: \"kubernetes.io/projected/ae8b5950-ddff-413c-946f-20b797585a14-kube-api-access-7hv56\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531957    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dfb0ebd2-1b79-45cb-893f-edabc9497184-iptables-alerter-script\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfb0ebd2-1b79-45cb-893f-edabc9497184-host-slash\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.531984    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2840b062-0201-4147-bc5e-17ceb2652dc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jhp\" (UniqueName: \"kubernetes.io/projected/dfb0ebd2-1b79-45cb-893f-edabc9497184-kube-api-access-g5jhp\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.533440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysctl-conf\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532089    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-socket-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-var-lib-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532140    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-os-release\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532168    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-sysctl-conf\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovnkube-script-lib\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-dbus\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5525z\" (UniqueName: \"kubernetes.io/projected/370616ca-420a-4679-91d6-9c035007c735-kube-api-access-5525z\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69srp\" (UniqueName: \"kubernetes.io/projected/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-kube-api-access-69srp\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532330    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-var-lib-openvswitch\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-systemd\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovn-node-metrics-cert\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-systemd-units\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532477    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532542    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-dbus\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532552    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-systemd-units\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-registration-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532588    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-registration-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e60e3631-e481-4969-9b02-2db258f40abc-serviceca\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532666    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8b5950-ddff-413c-946f-20b797585a14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532703    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-run-systemd\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532735    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovnkube-script-lib\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.534782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.532842    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2840b062-0201-4147-bc5e-17ceb2652dc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.535378 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.535357    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-tmp\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.535774 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.535721    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-ovn-node-metrics-cert\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.536804 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.536751    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a975ef5-c4da-4258-a07d-41d5259defde-agent-certs\") pod \"konnectivity-agent-h8snv\" (UID: \"1a975ef5-c4da-4258-a07d-41d5259defde\") " pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:21.537021 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.537003    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-etc-tuned\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.540572 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.540288    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktpg2\" (UniqueName: \"kubernetes.io/projected/12e4acd4-8b5f-40ae-b2cd-2399a39d43d8-kube-api-access-ktpg2\") pod \"ovnkube-node-8pcqm\" (UID: \"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.541531 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.541373    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfrn\" (UniqueName: \"kubernetes.io/projected/2840b062-0201-4147-bc5e-17ceb2652dc7-kube-api-access-jrfrn\") pod \"multus-additional-cni-plugins-bx962\" (UID: \"2840b062-0201-4147-bc5e-17ceb2652dc7\") " pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.542924 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.542879    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hv56\" (UniqueName: \"kubernetes.io/projected/ae8b5950-ddff-413c-946f-20b797585a14-kube-api-access-7hv56\") pod \"aws-ebs-csi-driver-node-bd2jb\" (UID: \"ae8b5950-ddff-413c-946f-20b797585a14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.544517 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.544475    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69srp\" (UniqueName: \"kubernetes.io/projected/9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41-kube-api-access-69srp\") pod \"tuned-vwl98\" (UID: \"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41\") " pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.607190 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.607132    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" event={"ID":"c73243dc8a0860bf821a5ab616d0670d","Type":"ContainerStarted","Data":"8c0f3e525b0dbee445089c18b876e8d67872567828241afe94562e54ad40bc3d"}
Jul 08 16:34:21.609849 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.609810    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"3f3918137afdb7026e67d79021b1461d","Type":"ContainerStarted","Data":"1a5221eb8fe2c804556bab7cb7ec53a7d94915b0252330c82451ee2a1ce8ca91"}
Jul 08 16:34:21.633376 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633351    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-cnibin\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633485 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-cni-bin\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633485 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-multus-certs\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633485 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633477    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-cnibin\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633492    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-cni-bin\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-k8s-cni-cncf-io\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633552    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-multus-certs\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633583    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-k8s-cni-cncf-io\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633593    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-etc-kubernetes\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-etc-kubernetes\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633635    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370616ca-420a-4679-91d6-9c035007c735-tmp-dir\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.633673 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.633674    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzccj\" (UniqueName: \"kubernetes.io/projected/8751bafd-f26c-47bd-8fa8-f24755c1fb18-kube-api-access-kzccj\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-cni-binary-copy\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.633721    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs podName:8751bafd-f26c-47bd-8fa8-f24755c1fb18 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:22.133710634 +0000 UTC m=+3.123267355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs") pod "network-metrics-daemon-4x4wh" (UID: "8751bafd-f26c-47bd-8fa8-f24755c1fb18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-hostroot\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-netns\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633799    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-hostroot\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-kubelet\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/370616ca-420a-4679-91d6-9c035007c735-hosts-file\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.633984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dfb0ebd2-1b79-45cb-893f-edabc9497184-iptables-alerter-script\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfb0ebd2-1b79-45cb-893f-edabc9497184-host-slash\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634012    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-kubelet\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jhp\" (UniqueName: \"kubernetes.io/projected/dfb0ebd2-1b79-45cb-893f-edabc9497184-kube-api-access-g5jhp\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:21.634091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634065    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370616ca-420a-4679-91d6-9c035007c735-tmp-dir\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5525z\" (UniqueName: \"kubernetes.io/projected/370616ca-420a-4679-91d6-9c035007c735-kube-api-access-5525z\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634112    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-run-netns\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634236    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-cni-binary-copy\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634241    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/370616ca-420a-4679-91d6-9c035007c735-hosts-file\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e60e3631-e481-4969-9b02-2db258f40abc-serviceca\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-cni-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634306    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfb0ebd2-1b79-45cb-893f-edabc9497184-host-slash\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-cni-multus\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634374    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-cni-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60e3631-e481-4969-9b02-2db258f40abc-host\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634409    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-system-cni-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634421    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-host-var-lib-cni-multus\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-os-release\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-socket-dir-parent\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.634830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634463    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60e3631-e481-4969-9b02-2db258f40abc-host\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634477    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-conf-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsz77\" (UniqueName: \"kubernetes.io/projected/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-kube-api-access-vsz77\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634525    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-os-release\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634538    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-daemon-config\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6p5\" (UniqueName: \"kubernetes.io/projected/e60e3631-e481-4969-9b02-2db258f40abc-kube-api-access-9f6p5\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634577    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-system-cni-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634664    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e60e3631-e481-4969-9b02-2db258f40abc-serviceca\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634695    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-conf-dir\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634713    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-socket-dir-parent\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.634765    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dfb0ebd2-1b79-45cb-893f-edabc9497184-iptables-alerter-script\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.635324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.635035    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-multus-daemon-config\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.646788 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.646293    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:21.646788 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.646319    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:21.646788 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.646333    2575 projected.go:196] Error preparing data for projected volume kube-api-access-tvkgc for pod openshift-network-diagnostics/network-check-target-dtcdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:21.646788 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:21.646403    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc podName:2f1fa945-7def-4e82-9da9-78010a748562 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:22.146385644 +0000 UTC m=+3.135942371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tvkgc" (UniqueName: "kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc") pod "network-check-target-dtcdw" (UID: "2f1fa945-7def-4e82-9da9-78010a748562") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:21.647057 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.646918    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5525z\" (UniqueName: \"kubernetes.io/projected/370616ca-420a-4679-91d6-9c035007c735-kube-api-access-5525z\") pod \"node-resolver-vtcrd\" (UID: \"370616ca-420a-4679-91d6-9c035007c735\") " pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.647257 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.647213    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzccj\" (UniqueName: \"kubernetes.io/projected/8751bafd-f26c-47bd-8fa8-f24755c1fb18-kube-api-access-kzccj\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:21.647359 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.647284    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jhp\" (UniqueName: \"kubernetes.io/projected/dfb0ebd2-1b79-45cb-893f-edabc9497184-kube-api-access-g5jhp\") pod \"iptables-alerter-gjvrt\" (UID: \"dfb0ebd2-1b79-45cb-893f-edabc9497184\") " pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.649504 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.649478    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6p5\" (UniqueName: \"kubernetes.io/projected/e60e3631-e481-4969-9b02-2db258f40abc-kube-api-access-9f6p5\") pod \"node-ca-8p82f\" (UID: \"e60e3631-e481-4969-9b02-2db258f40abc\") " pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.649504 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.649487    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsz77\" (UniqueName: \"kubernetes.io/projected/81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6-kube-api-access-vsz77\") pod \"multus-6bqfr\" (UID: \"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6\") " pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.715741 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.715720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vwl98"
Jul 08 16:34:21.723449 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.723428    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:21.732939 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.732908    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:21.740396 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.740377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bx962"
Jul 08 16:34:21.747839 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.747820    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb"
Jul 08 16:34:21.758196 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.758167    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8p82f"
Jul 08 16:34:21.765709 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.765692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6bqfr"
Jul 08 16:34:21.774236 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.774220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gjvrt"
Jul 08 16:34:21.779727 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.779614    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vtcrd"
Jul 08 16:34:21.895760 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:21.895735    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:22.037156 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:22.037088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:22.037286 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.037210    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:22.037286 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.037257    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret podName:2a323b89-9740-4ab1-9d0d-1ca2bec3d93d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:23.037244176 +0000 UTC m=+4.026800897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret") pod "global-pull-secret-syncer-v8x8z" (UID: "2a323b89-9740-4ab1-9d0d-1ca2bec3d93d") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:22.138276 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:22.138250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:22.138416 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.138368    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:22.138456 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.138417    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs podName:8751bafd-f26c-47bd-8fa8-f24755c1fb18 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:23.138403985 +0000 UTC m=+4.127960705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs") pod "network-metrics-daemon-4x4wh" (UID: "8751bafd-f26c-47bd-8fa8-f24755c1fb18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:22.239078 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:22.239052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:22.239248 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.239196    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:22.239248 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.239219    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:22.239248 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.239231    2575 projected.go:196] Error preparing data for projected volume kube-api-access-tvkgc for pod openshift-network-diagnostics/network-check-target-dtcdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:22.239384 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.239294    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc podName:2f1fa945-7def-4e82-9da9-78010a748562 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:23.239274904 +0000 UTC m=+4.228831648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvkgc" (UniqueName: "kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc") pod "network-check-target-dtcdw" (UID: "2f1fa945-7def-4e82-9da9-78010a748562") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:22.464796 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:22.464760    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:21 +0000 UTC" deadline="2027-12-03 14:56:23.514919258 +0000 UTC"
Jul 08 16:34:22.464796 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:22.464794    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12310h22m1.050130262s"
Jul 08 16:34:22.603225 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:22.603197    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:22.603366 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:22.603307    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:22.684668 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.684630    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a975ef5_c4da_4258_a07d_41d5259defde.slice/crio-93182c2906f5e126239b0ac4d292f65926a70d7a17e3b64905aa404da6966849 WatchSource:0}: Error finding container 93182c2906f5e126239b0ac4d292f65926a70d7a17e3b64905aa404da6966849: Status 404 returned error can't find the container with id 93182c2906f5e126239b0ac4d292f65926a70d7a17e3b64905aa404da6966849
Jul 08 16:34:22.686584 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.686549    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb0ebd2_1b79_45cb_893f_edabc9497184.slice/crio-8b07b34c25af602ee5379ba4c36c6136c895905c193e85c865b914fc25cbc96a WatchSource:0}: Error finding container 8b07b34c25af602ee5379ba4c36c6136c895905c193e85c865b914fc25cbc96a: Status 404 returned error can't find the container with id 8b07b34c25af602ee5379ba4c36c6136c895905c193e85c865b914fc25cbc96a
Jul 08 16:34:22.687350 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.687326    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e4acd4_8b5f_40ae_b2cd_2399a39d43d8.slice/crio-0dc4a803579703a4bb23234698e42bcdf6e5ad07950e9c2c434ad6605892d037 WatchSource:0}: Error finding container 0dc4a803579703a4bb23234698e42bcdf6e5ad07950e9c2c434ad6605892d037: Status 404 returned error can't find the container with id 0dc4a803579703a4bb23234698e42bcdf6e5ad07950e9c2c434ad6605892d037
Jul 08 16:34:22.690046 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.690025    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2840b062_0201_4147_bc5e_17ceb2652dc7.slice/crio-7aa9e6dde8218e9ca9966eecc2b29efa55d799705817df0a1accc118817fa53c WatchSource:0}: Error finding container 7aa9e6dde8218e9ca9966eecc2b29efa55d799705817df0a1accc118817fa53c: Status 404 returned error can't find the container with id 7aa9e6dde8218e9ca9966eecc2b29efa55d799705817df0a1accc118817fa53c
Jul 08 16:34:22.691222 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.690941    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370616ca_420a_4679_91d6_9c035007c735.slice/crio-907b1635ecb5d475c41825a3ede7daa44f7bf8e096a794df8176df29b836e693 WatchSource:0}: Error finding container 907b1635ecb5d475c41825a3ede7daa44f7bf8e096a794df8176df29b836e693: Status 404 returned error can't find the container with id 907b1635ecb5d475c41825a3ede7daa44f7bf8e096a794df8176df29b836e693
Jul 08 16:34:22.692665 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.692623    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e545bb_0f25_4af4_b5fe_6ef6ba0edcd6.slice/crio-65d83f47887d04ecaa2b27b4bceec7af9a0475849f95fde33c28db897af2a0b0 WatchSource:0}: Error finding container 65d83f47887d04ecaa2b27b4bceec7af9a0475849f95fde33c28db897af2a0b0: Status 404 returned error can't find the container with id 65d83f47887d04ecaa2b27b4bceec7af9a0475849f95fde33c28db897af2a0b0
Jul 08 16:34:22.693963 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.693704    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0d5c2b_438f_44a5_a6e0_9a9c70b94b41.slice/crio-edf9be57834646e18f3b65b0b89b5c73e3d62d6b33b3bf625f5165c235189750 WatchSource:0}: Error finding container edf9be57834646e18f3b65b0b89b5c73e3d62d6b33b3bf625f5165c235189750: Status 404 returned error can't find the container with id edf9be57834646e18f3b65b0b89b5c73e3d62d6b33b3bf625f5165c235189750
Jul 08 16:34:22.695769 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.695721    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8b5950_ddff_413c_946f_20b797585a14.slice/crio-7e2a4f4121b55991652bd40c236e6e1a418fd433b0ae6a3b1d60d639ec470c27 WatchSource:0}: Error finding container 7e2a4f4121b55991652bd40c236e6e1a418fd433b0ae6a3b1d60d639ec470c27: Status 404 returned error can't find the container with id 7e2a4f4121b55991652bd40c236e6e1a418fd433b0ae6a3b1d60d639ec470c27
Jul 08 16:34:22.699024 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:22.699002    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60e3631_e481_4969_9b02_2db258f40abc.slice/crio-1783c83d0e943e2eda097c5cd8baffa6364d107ba9247e6d63cd2f153f8175c2 WatchSource:0}: Error finding container 1783c83d0e943e2eda097c5cd8baffa6364d107ba9247e6d63cd2f153f8175c2: Status 404 returned error can't find the container with id 1783c83d0e943e2eda097c5cd8baffa6364d107ba9247e6d63cd2f153f8175c2
Jul 08 16:34:23.045140 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.045073    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:23.045225 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.045207    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:23.045263 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.045257    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret podName:2a323b89-9740-4ab1-9d0d-1ca2bec3d93d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:25.045238162 +0000 UTC m=+6.034794882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret") pod "global-pull-secret-syncer-v8x8z" (UID: "2a323b89-9740-4ab1-9d0d-1ca2bec3d93d") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:23.145946 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.145918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:23.146108 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.146079    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:23.146221 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.146145    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs podName:8751bafd-f26c-47bd-8fa8-f24755c1fb18 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:25.14612625 +0000 UTC m=+6.135682974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs") pod "network-metrics-daemon-4x4wh" (UID: "8751bafd-f26c-47bd-8fa8-f24755c1fb18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:23.246404 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.246372    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:23.246990 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.246564    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:23.246990 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.246589    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:23.246990 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.246603    2575 projected.go:196] Error preparing data for projected volume kube-api-access-tvkgc for pod openshift-network-diagnostics/network-check-target-dtcdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:23.246990 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.246675    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc podName:2f1fa945-7def-4e82-9da9-78010a748562 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:25.246636931 +0000 UTC m=+6.236193672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvkgc" (UniqueName: "kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc") pod "network-check-target-dtcdw" (UID: "2f1fa945-7def-4e82-9da9-78010a748562") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:23.465795 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.465725    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:21 +0000 UTC" deadline="2028-02-25 19:21:38.616730633 +0000 UTC"
Jul 08 16:34:23.465795 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.465768    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14330h47m15.150967591s"
Jul 08 16:34:23.612026 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.604231    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:23.612026 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.604436    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:23.612026 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.611827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:23.612026 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:23.611967    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:23.636106 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.635503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" event={"ID":"c73243dc8a0860bf821a5ab616d0670d","Type":"ContainerStarted","Data":"ca29cc57851aa403f0acabb7b8c33f8623979e88f73e3560ab981944c5a6ce35"}
Jul 08 16:34:23.650238 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.650114    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vwl98" event={"ID":"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41","Type":"ContainerStarted","Data":"edf9be57834646e18f3b65b0b89b5c73e3d62d6b33b3bf625f5165c235189750"}
Jul 08 16:34:23.666159 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.666128    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6bqfr" event={"ID":"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6","Type":"ContainerStarted","Data":"65d83f47887d04ecaa2b27b4bceec7af9a0475849f95fde33c28db897af2a0b0"}
Jul 08 16:34:23.672419 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.672393    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8p82f" event={"ID":"e60e3631-e481-4969-9b02-2db258f40abc","Type":"ContainerStarted","Data":"1783c83d0e943e2eda097c5cd8baffa6364d107ba9247e6d63cd2f153f8175c2"}
Jul 08 16:34:23.688325 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.687938    2575 generic.go:356] "Generic (PLEG): container finished" podID="3f3918137afdb7026e67d79021b1461d" containerID="379bfd38ebda733aae2488d5c6407e99446a87626b9a7853f807441729a5e0ed" exitCode=0
Jul 08 16:34:23.688325 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.688022    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"3f3918137afdb7026e67d79021b1461d","Type":"ContainerDied","Data":"379bfd38ebda733aae2488d5c6407e99446a87626b9a7853f807441729a5e0ed"}
Jul 08 16:34:23.702124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.701440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vtcrd" event={"ID":"370616ca-420a-4679-91d6-9c035007c735","Type":"ContainerStarted","Data":"907b1635ecb5d475c41825a3ede7daa44f7bf8e096a794df8176df29b836e693"}
Jul 08 16:34:23.703710 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.703662    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" podStartSLOduration=3.703624884 podStartE2EDuration="3.703624884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:23.652883176 +0000 UTC m=+4.642439919" watchObservedRunningTime="2026-07-08 16:34:23.703624884 +0000 UTC m=+4.693181631"
Jul 08 16:34:23.719313 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.719288    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb" event={"ID":"ae8b5950-ddff-413c-946f-20b797585a14","Type":"ContainerStarted","Data":"7e2a4f4121b55991652bd40c236e6e1a418fd433b0ae6a3b1d60d639ec470c27"}
Jul 08 16:34:23.726693 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.726606    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bx962" event={"ID":"2840b062-0201-4147-bc5e-17ceb2652dc7","Type":"ContainerStarted","Data":"7aa9e6dde8218e9ca9966eecc2b29efa55d799705817df0a1accc118817fa53c"}
Jul 08 16:34:23.732055 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.731963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"0dc4a803579703a4bb23234698e42bcdf6e5ad07950e9c2c434ad6605892d037"}
Jul 08 16:34:23.745858 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.745833    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gjvrt" event={"ID":"dfb0ebd2-1b79-45cb-893f-edabc9497184","Type":"ContainerStarted","Data":"8b07b34c25af602ee5379ba4c36c6136c895905c193e85c865b914fc25cbc96a"}
Jul 08 16:34:23.751987 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:23.751964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h8snv" event={"ID":"1a975ef5-c4da-4258-a07d-41d5259defde","Type":"ContainerStarted","Data":"93182c2906f5e126239b0ac4d292f65926a70d7a17e3b64905aa404da6966849"}
Jul 08 16:34:24.602923 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:24.602893    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:24.603383 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:24.603016    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:24.766065 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:24.765997    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"3f3918137afdb7026e67d79021b1461d","Type":"ContainerStarted","Data":"8d905b029b7201584713a7b011b2dfef6c242806bf069a6b9737082f7f868660"}
Jul 08 16:34:25.062484 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:25.062399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:25.062662 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.062558    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:25.062662 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.062623    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret podName:2a323b89-9740-4ab1-9d0d-1ca2bec3d93d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:29.062604312 +0000 UTC m=+10.052161051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret") pod "global-pull-secret-syncer-v8x8z" (UID: "2a323b89-9740-4ab1-9d0d-1ca2bec3d93d") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:25.163391 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:25.162770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:25.163391 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.162964    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:25.163391 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.163074    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs podName:8751bafd-f26c-47bd-8fa8-f24755c1fb18 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:29.163054372 +0000 UTC m=+10.152611097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs") pod "network-metrics-daemon-4x4wh" (UID: "8751bafd-f26c-47bd-8fa8-f24755c1fb18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:25.263731 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:25.263695    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:25.263933 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.263918    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:25.264016 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.263939    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:25.264016 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.263951    2575 projected.go:196] Error preparing data for projected volume kube-api-access-tvkgc for pod openshift-network-diagnostics/network-check-target-dtcdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:25.264016 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.264005    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc podName:2f1fa945-7def-4e82-9da9-78010a748562 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:29.263987366 +0000 UTC m=+10.253544103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvkgc" (UniqueName: "kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc") pod "network-check-target-dtcdw" (UID: "2f1fa945-7def-4e82-9da9-78010a748562") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:25.606250 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:25.605472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:25.606250 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.605628    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:25.606250 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:25.605718    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:25.606250 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:25.605802    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:26.603005 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:26.602977    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:26.603175 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:26.603104    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:27.603244 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:27.603193    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:27.603701 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:27.603342    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:27.603701 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:27.603418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:27.603701 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:27.603566    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:28.603062 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:28.603031    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:28.603249 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:28.603162    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:29.096519 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:29.096438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:29.096976 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.096628    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:29.096976 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.096723    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret podName:2a323b89-9740-4ab1-9d0d-1ca2bec3d93d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:37.096699453 +0000 UTC m=+18.086256176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret") pod "global-pull-secret-syncer-v8x8z" (UID: "2a323b89-9740-4ab1-9d0d-1ca2bec3d93d") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:29.197105 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:29.196864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:29.197105 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.196995    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:29.197105 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.197062    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs podName:8751bafd-f26c-47bd-8fa8-f24755c1fb18 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:37.197044067 +0000 UTC m=+18.186600805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs") pod "network-metrics-daemon-4x4wh" (UID: "8751bafd-f26c-47bd-8fa8-f24755c1fb18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:29.297558 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:29.297505    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:29.297737 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.297718    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:29.297815 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.297746    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:29.297815 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.297760    2575 projected.go:196] Error preparing data for projected volume kube-api-access-tvkgc for pod openshift-network-diagnostics/network-check-target-dtcdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:29.297909 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.297825    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc podName:2f1fa945-7def-4e82-9da9-78010a748562 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:37.29780549 +0000 UTC m=+18.287362224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvkgc" (UniqueName: "kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc") pod "network-check-target-dtcdw" (UID: "2f1fa945-7def-4e82-9da9-78010a748562") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:29.604575 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:29.604543    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:29.604767 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.604703    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:29.605062 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:29.605046    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:29.605168 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:29.605147    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:30.603016 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:30.602986    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:30.603470 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:30.603113    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:31.603216 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:31.603064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:31.603216 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:31.603063    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:31.603216 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:31.603202    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:31.603805 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:31.603299    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:32.603167 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:32.603140    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:32.603305 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:32.603237    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:33.603761 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:33.603731    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:33.603761 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:33.603761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:33.604192 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:33.603872    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:33.604192 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:33.603949    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:34.603297 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:34.603259    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:34.603459 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:34.603385    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:35.602859 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:35.602832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:35.603284 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:35.602843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:35.603284 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:35.602951    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:35.603284 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:35.603012    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:36.603759 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:36.603731    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:36.604187 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:36.603825    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:37.157444 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:37.157409    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:37.157589 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.157540    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:37.157628 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.157602    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret podName:2a323b89-9740-4ab1-9d0d-1ca2bec3d93d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:53.157582012 +0000 UTC m=+34.147138734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret") pod "global-pull-secret-syncer-v8x8z" (UID: "2a323b89-9740-4ab1-9d0d-1ca2bec3d93d") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:37.257701 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:37.257675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:37.257838 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.257769    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:37.257838 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.257814    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs podName:8751bafd-f26c-47bd-8fa8-f24755c1fb18 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:53.257800631 +0000 UTC m=+34.247357351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs") pod "network-metrics-daemon-4x4wh" (UID: "8751bafd-f26c-47bd-8fa8-f24755c1fb18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:37.358416 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:37.358386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:37.358523 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.358513    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:37.358578 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.358532    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:37.358578 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.358545    2575 projected.go:196] Error preparing data for projected volume kube-api-access-tvkgc for pod openshift-network-diagnostics/network-check-target-dtcdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:37.358716 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.358601    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc podName:2f1fa945-7def-4e82-9da9-78010a748562 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:53.358584568 +0000 UTC m=+34.348141296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvkgc" (UniqueName: "kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc") pod "network-check-target-dtcdw" (UID: "2f1fa945-7def-4e82-9da9-78010a748562") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:37.602778 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:37.602757    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:37.602863 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:37.602786    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:37.602863 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.602850    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:37.602977 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:37.602959    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:38.603721 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:38.603687    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:38.604177 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:38.603814    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:39.604336 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:39.604304    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:39.604789 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:39.604411    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:39.604789 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:39.604519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:39.604789 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:39.604668    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:40.603633 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:40.603601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:40.603798 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:40.603710    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:41.606548 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.605710    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:41.606548 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:41.606004    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:41.606937 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.606555    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:41.606937 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:41.606698    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:41.804262 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.804234    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h8snv" event={"ID":"1a975ef5-c4da-4258-a07d-41d5259defde","Type":"ContainerStarted","Data":"9cbd0bf9d567a627e743ca375b190b9d128c9be80b74b44f87a9d0b47aa1c16c"}
Jul 08 16:34:41.805817 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.805793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vwl98" event={"ID":"9c0d5c2b-438f-44a5-a6e0-9a9c70b94b41","Type":"ContainerStarted","Data":"4db6368539134c772e49d5334a28033c72f6cd617c6dcd4582e9556d3c6e3dd4"}
Jul 08 16:34:41.807421 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.807399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6bqfr" event={"ID":"81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6","Type":"ContainerStarted","Data":"a477ca841d3b3546afa20d5ee4e2e8b8d80e4573b251d4ca2ae7d61237d1e8ed"}
Jul 08 16:34:41.812455 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.812427    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8p82f" event={"ID":"e60e3631-e481-4969-9b02-2db258f40abc","Type":"ContainerStarted","Data":"254ba38decec2c5c6a9161ef271bc7d7265e2b05ea32bad051f9ac7caadf775d"}
Jul 08 16:34:41.813996 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.813971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vtcrd" event={"ID":"370616ca-420a-4679-91d6-9c035007c735","Type":"ContainerStarted","Data":"af14dd69c89943cd49e1ac3a5877969fcad1282c7a236196923251af271b8b27"}
Jul 08 16:34:41.815358 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.815337    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb" event={"ID":"ae8b5950-ddff-413c-946f-20b797585a14","Type":"ContainerStarted","Data":"725684050f9346bd23ca8263d11754269e4475d200b86bc44c8dc390828c70a2"}
Jul 08 16:34:41.816751 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.816730    2575 generic.go:356] "Generic (PLEG): container finished" podID="2840b062-0201-4147-bc5e-17ceb2652dc7" containerID="c3bd35cd0d2d6edfd15ad6f02352415874384c9b50579a85a13093f0aa6d6158" exitCode=0
Jul 08 16:34:41.816833 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.816762    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bx962" event={"ID":"2840b062-0201-4147-bc5e-17ceb2652dc7","Type":"ContainerDied","Data":"c3bd35cd0d2d6edfd15ad6f02352415874384c9b50579a85a13093f0aa6d6158"}
Jul 08 16:34:41.820524 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.820473    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h8snv" podStartSLOduration=4.179958842 podStartE2EDuration="22.820462762s" totalImagesPullingTime="18.64050392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:41.820362671 +0000 UTC m=+22.809919431" watchObservedRunningTime="2026-07-08 16:34:41.820462762 +0000 UTC m=+22.810019504"
Jul 08 16:34:41.820625 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.820548    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" podStartSLOduration=21.820543239 podStartE2EDuration="21.820543239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:24.782904079 +0000 UTC m=+5.772460826" watchObservedRunningTime="2026-07-08 16:34:41.820543239 +0000 UTC m=+22.810099981"
Jul 08 16:34:41.837305 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.837254    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vwl98" podStartSLOduration=4.15699488 podStartE2EDuration="22.837236936s" totalImagesPullingTime="18.680242056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:41.83638423 +0000 UTC m=+22.825940975" watchObservedRunningTime="2026-07-08 16:34:41.837236936 +0000 UTC m=+22.826793769"
Jul 08 16:34:41.872189 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.872151    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8p82f" podStartSLOduration=4.197211995 podStartE2EDuration="22.872138423s" totalImagesPullingTime="18.674926428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:41.87191381 +0000 UTC m=+22.861470588" watchObservedRunningTime="2026-07-08 16:34:41.872138423 +0000 UTC m=+22.861695165"
Jul 08 16:34:41.885912 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.885875    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vtcrd" podStartSLOduration=4.198062033 podStartE2EDuration="22.885862074s" totalImagesPullingTime="18.687800041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:41.885661475 +0000 UTC m=+22.875218247" watchObservedRunningTime="2026-07-08 16:34:41.885862074 +0000 UTC m=+22.875418817"
Jul 08 16:34:41.904508 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:41.904472    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6bqfr" podStartSLOduration=3.987465471 podStartE2EDuration="22.904460195s" totalImagesPullingTime="18.916994724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:41.904306633 +0000 UTC m=+22.893863379" watchObservedRunningTime="2026-07-08 16:34:41.904460195 +0000 UTC m=+22.894016938"
Jul 08 16:34:42.053023 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.052914    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:42.155123 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.155094    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:42.603462 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.603437    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:42.603624 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:42.603559    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:42.829070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.829011    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:34:42.829852 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.829398    2575 generic.go:356] "Generic (PLEG): container finished" podID="12e4acd4-8b5f-40ae-b2cd-2399a39d43d8" containerID="ef71571d501309fe299b296c94578c89a1e7fe33f24f7842db633d861ca7ca1d" exitCode=1
Jul 08 16:34:42.829852 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.829476    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"39db59601a6706f6feab8c099ba624d580c31ca64021376a1da52ad37b450b38"}
Jul 08 16:34:42.829852 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.829589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"a79c4d8a3b894b31a67ca01c6f630ea1eff51917ccd97941bf8dbfce29d16a59"}
Jul 08 16:34:42.829852 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.829633    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"729a8aa0ad21207566728f54515342c0404c108cb2609c3da4ef2bf86778caf3"}
Jul 08 16:34:42.829852 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.829701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"dad2c50417e8f1ae3806bcf57bd7d5e509f9a8b915835f1b2916270062226e83"}
Jul 08 16:34:42.829852 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.829748    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerDied","Data":"ef71571d501309fe299b296c94578c89a1e7fe33f24f7842db633d861ca7ca1d"}
Jul 08 16:34:42.829852 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.829793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"ba224df540457269db169a14708c6f92aec17ec36ac698c5bcd350fd79212663"}
Jul 08 16:34:42.831132 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.831099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gjvrt" event={"ID":"dfb0ebd2-1b79-45cb-893f-edabc9497184","Type":"ContainerStarted","Data":"0cb7c68e1847127c89e287abe4e89b9e23c56381f9863c918b075ec1f34f476d"}
Jul 08 16:34:42.848458 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.848404    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gjvrt" podStartSLOduration=5.162108088 podStartE2EDuration="23.848389938s" totalImagesPullingTime="18.68628185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:42.848127112 +0000 UTC m=+23.837683855" watchObservedRunningTime="2026-07-08 16:34:42.848389938 +0000 UTC m=+23.837946680"
Jul 08 16:34:42.972997 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:42.972966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:43.079422 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:43.079360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:43.094559 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:43.094538    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 16:34:43.474093 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:43.473959    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T16:34:43.094555266Z","UUID":"02ccb8fe-f3ac-4386-b2b0-2561051e1d0f","Handler":null,"Name":"","Endpoint":""}
Jul 08 16:34:43.475488 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:43.475470    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 16:34:43.475488 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:43.475502    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 16:34:43.603007 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:43.602979    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:43.603007 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:43.602984    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:43.603285 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:43.603113    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:43.603285 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:43.603201    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:43.836140 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:43.836057    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb" event={"ID":"ae8b5950-ddff-413c-946f-20b797585a14","Type":"ContainerStarted","Data":"2e32653ab3b41b3bcff43c41b61b8498d1a8fddac4b23043b81ad846a451b424"}
Jul 08 16:34:44.385738 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:44.385688    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:44.524307 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:44.524269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:44.603070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:44.603045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:44.603206 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:44.603147    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:45.603583 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:45.603387    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:45.604038 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:45.603456    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:45.604038 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:45.603694    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:45.604038 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:45.603757    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:46.242182 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.242130    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:46.242777 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.242762    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:46.459012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.458956    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:46.459585 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.459569    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h8snv"
Jul 08 16:34:46.525622 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.525595    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:46.603300 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.603278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:46.603392 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:46.603375    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:46.842693 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.842612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb" event={"ID":"ae8b5950-ddff-413c-946f-20b797585a14","Type":"ContainerStarted","Data":"08eb253a440818fad169f253dd45351ecd80a405dfce3977a31767681a704766"}
Jul 08 16:34:46.844527 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.844503    2575 generic.go:356] "Generic (PLEG): container finished" podID="2840b062-0201-4147-bc5e-17ceb2652dc7" containerID="b5b43cbd25c6d05d06c160863a8a5f42c6bc20d520932cd6861f44f19de44546" exitCode=0
Jul 08 16:34:46.844610 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.844591    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bx962" event={"ID":"2840b062-0201-4147-bc5e-17ceb2652dc7","Type":"ContainerDied","Data":"b5b43cbd25c6d05d06c160863a8a5f42c6bc20d520932cd6861f44f19de44546"}
Jul 08 16:34:46.848183 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.848165    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:34:46.848603 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.848582    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"6f3ea491ce02379cb2aa575738bdef06b8c857d11a4cc9717de479087de1dd01"}
Jul 08 16:34:46.860786 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:46.860750    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd2jb" podStartSLOduration=4.804875914 podStartE2EDuration="27.860734033s" totalImagesPullingTime="23.055858119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:46.86030019 +0000 UTC m=+27.849856932" watchObservedRunningTime="2026-07-08 16:34:46.860734033 +0000 UTC m=+27.850290778"
Jul 08 16:34:47.603102 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:47.603071    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:47.603357 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:47.603177    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:47.603357 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:47.603243    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:47.603453 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:47.603358    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:48.603265 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.603121    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:48.603696 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:48.603341    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:48.854146 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.854069    2575 generic.go:356] "Generic (PLEG): container finished" podID="2840b062-0201-4147-bc5e-17ceb2652dc7" containerID="7bbdf7e690f9e8261a1195231dbdf280ec2f46bece2301050fc689e3ade16f5d" exitCode=0
Jul 08 16:34:48.854299 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.854159    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bx962" event={"ID":"2840b062-0201-4147-bc5e-17ceb2652dc7","Type":"ContainerDied","Data":"7bbdf7e690f9e8261a1195231dbdf280ec2f46bece2301050fc689e3ade16f5d"}
Jul 08 16:34:48.857385 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.857368    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:34:48.857749 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.857725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"310511d7b1a23fe39cf86cbcc3f94c250ba2e75a030eea56e5adae657e5c5adb"}
Jul 08 16:34:48.857945 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.857928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:48.858016 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.857989    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:48.858187 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.858174    2575 scope.go:117] "RemoveContainer" containerID="ef71571d501309fe299b296c94578c89a1e7fe33f24f7842db633d861ca7ca1d"
Jul 08 16:34:48.873502 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:48.873477    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:49.603633 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:49.603597    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:49.604234 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:49.603780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:49.604234 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:49.603856    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:49.604234 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:49.603750    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:49.865735 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:49.865681    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:34:49.866072 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:49.866046    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" event={"ID":"12e4acd4-8b5f-40ae-b2cd-2399a39d43d8","Type":"ContainerStarted","Data":"769bfecf067c2c4915bd6109491185a9c02ae53c4a572fa3944143591145b138"}
Jul 08 16:34:49.866248 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:49.866232    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:49.882291 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:49.882268    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:34:49.895029 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:49.894992    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm" podStartSLOduration=11.860729963 podStartE2EDuration="30.894979792s" totalImagesPullingTime="19.034249829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:49.893324011 +0000 UTC m=+30.882880754" watchObservedRunningTime="2026-07-08 16:34:49.894979792 +0000 UTC m=+30.884536536"
Jul 08 16:34:50.562296 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:50.562127    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v8x8z"]
Jul 08 16:34:50.562453 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:50.562396    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:50.562550 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:50.562491    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:50.565516 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:50.565490    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4x4wh"]
Jul 08 16:34:50.565690 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:50.565588    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:50.565798 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:50.565763    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:50.566169 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:50.566120    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dtcdw"]
Jul 08 16:34:50.566265 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:50.566216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:50.566319 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:50.566289    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:50.870750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:50.870714    2575 generic.go:356] "Generic (PLEG): container finished" podID="2840b062-0201-4147-bc5e-17ceb2652dc7" containerID="001856a3f7bac8b27f733dbf17361505e7f23fc28a9c2dc939a168e43cf3c7c8" exitCode=0
Jul 08 16:34:50.871089 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:50.870812    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bx962" event={"ID":"2840b062-0201-4147-bc5e-17ceb2652dc7","Type":"ContainerDied","Data":"001856a3f7bac8b27f733dbf17361505e7f23fc28a9c2dc939a168e43cf3c7c8"}
Jul 08 16:34:51.512146 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:51.512113    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:51.613272 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:51.613234    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:52.603175 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:52.603143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:52.603821 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:52.603264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:52.603821 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:52.603270    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:52.603821 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:52.603377    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:52.603821 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:52.603271    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:52.603821 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:52.603467    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:53.000876 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:53.000787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:53.102244 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:53.102199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:53.191059 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:53.191024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:53.191207 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:53.191168    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:53.191273 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:53.191241    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret podName:2a323b89-9740-4ab1-9d0d-1ca2bec3d93d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:25.191221932 +0000 UTC m=+66.180778667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret") pod "global-pull-secret-syncer-v8x8z" (UID: "2a323b89-9740-4ab1-9d0d-1ca2bec3d93d") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:53.291701 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:53.291620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:53.291819 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:53.291765    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:53.291860 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:53.291832    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs podName:8751bafd-f26c-47bd-8fa8-f24755c1fb18 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:25.291811625 +0000 UTC m=+66.281368357 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs") pod "network-metrics-daemon-4x4wh" (UID: "8751bafd-f26c-47bd-8fa8-f24755c1fb18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:53.391945 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:53.391918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:53.392085 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:53.392065    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:53.392085 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:53.392085    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:53.392170 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:53.392096    2575 projected.go:196] Error preparing data for projected volume kube-api-access-tvkgc for pod openshift-network-diagnostics/network-check-target-dtcdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:53.392170 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:53.392156    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc podName:2f1fa945-7def-4e82-9da9-78010a748562 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:25.392138974 +0000 UTC m=+66.381695775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvkgc" (UniqueName: "kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc") pod "network-check-target-dtcdw" (UID: "2f1fa945-7def-4e82-9da9-78010a748562") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:54.603308 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:54.603276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:54.603868 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:54.603319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:54.603868 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:54.603401    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:54.603868 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:54.603413    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x4wh" podUID="8751bafd-f26c-47bd-8fa8-f24755c1fb18"
Jul 08 16:34:54.603868 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:54.603513    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8x8z" podUID="2a323b89-9740-4ab1-9d0d-1ca2bec3d93d"
Jul 08 16:34:54.603868 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:54.603635    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtcdw" podUID="2f1fa945-7def-4e82-9da9-78010a748562"
Jul 08 16:34:55.237573 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.237554    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeReady"
Jul 08 16:34:55.237667 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.237662    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 16:34:55.284686 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.284351    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p8lx6"]
Jul 08 16:34:55.307276 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.307259    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-klkjg"]
Jul 08 16:34:55.307392 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.307376    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.309538 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.309519    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 16:34:55.310291 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.310275    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jflkn\""
Jul 08 16:34:55.310346 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.310316    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 16:34:55.323251 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.323234    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p8lx6"]
Jul 08 16:34:55.323356 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.323272    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-klkjg"]
Jul 08 16:34:55.323410 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.323396    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-klkjg"
Jul 08 16:34:55.326081 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.326016    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 16:34:55.326180 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.326108    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 16:34:55.329173 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.327621    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lgg5h\""
Jul 08 16:34:55.329173 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.327912    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 16:34:55.406992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.406963    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0dfd2e0-9a06-4c80-b124-7a84a7e92543-cert\") pod \"ingress-canary-klkjg\" (UID: \"f0dfd2e0-9a06-4c80-b124-7a84a7e92543\") " pod="openshift-ingress-canary/ingress-canary-klkjg"
Jul 08 16:34:55.407100 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.406996    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.407100 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.407013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa748963-12ef-4075-8741-ab58379b8a60-tmp-dir\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.407100 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.407032    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2fl\" (UniqueName: \"kubernetes.io/projected/fa748963-12ef-4075-8741-ab58379b8a60-kube-api-access-7h2fl\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.407100 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.407054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa748963-12ef-4075-8741-ab58379b8a60-config-volume\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.407232 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.407143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnvbb\" (UniqueName: \"kubernetes.io/projected/f0dfd2e0-9a06-4c80-b124-7a84a7e92543-kube-api-access-gnvbb\") pod \"ingress-canary-klkjg\" (UID: \"f0dfd2e0-9a06-4c80-b124-7a84a7e92543\") " pod="openshift-ingress-canary/ingress-canary-klkjg"
Jul 08 16:34:55.508029 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.507975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvbb\" (UniqueName: \"kubernetes.io/projected/f0dfd2e0-9a06-4c80-b124-7a84a7e92543-kube-api-access-gnvbb\") pod \"ingress-canary-klkjg\" (UID: \"f0dfd2e0-9a06-4c80-b124-7a84a7e92543\") " pod="openshift-ingress-canary/ingress-canary-klkjg"
Jul 08 16:34:55.508125 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.508031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0dfd2e0-9a06-4c80-b124-7a84a7e92543-cert\") pod \"ingress-canary-klkjg\" (UID: \"f0dfd2e0-9a06-4c80-b124-7a84a7e92543\") " pod="openshift-ingress-canary/ingress-canary-klkjg"
Jul 08 16:34:55.508125 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.508047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.508125 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.508062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa748963-12ef-4075-8741-ab58379b8a60-tmp-dir\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.508125 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.508081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2fl\" (UniqueName: \"kubernetes.io/projected/fa748963-12ef-4075-8741-ab58379b8a60-kube-api-access-7h2fl\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.508125 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.508097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa748963-12ef-4075-8741-ab58379b8a60-config-volume\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.508383 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:55.508183    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:55.508383 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:55.508248    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls podName:fa748963-12ef-4075-8741-ab58379b8a60 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:56.008231336 +0000 UTC m=+36.997788077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls") pod "dns-default-p8lx6" (UID: "fa748963-12ef-4075-8741-ab58379b8a60") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:55.508576 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.508556    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa748963-12ef-4075-8741-ab58379b8a60-tmp-dir\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.508698 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.508680    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa748963-12ef-4075-8741-ab58379b8a60-config-volume\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.512142 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.512126    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0dfd2e0-9a06-4c80-b124-7a84a7e92543-cert\") pod \"ingress-canary-klkjg\" (UID: \"f0dfd2e0-9a06-4c80-b124-7a84a7e92543\") " pod="openshift-ingress-canary/ingress-canary-klkjg"
Jul 08 16:34:55.516749 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.516731    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2fl\" (UniqueName: \"kubernetes.io/projected/fa748963-12ef-4075-8741-ab58379b8a60-kube-api-access-7h2fl\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:55.516828 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.516760    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvbb\" (UniqueName: \"kubernetes.io/projected/f0dfd2e0-9a06-4c80-b124-7a84a7e92543-kube-api-access-gnvbb\") pod \"ingress-canary-klkjg\" (UID: \"f0dfd2e0-9a06-4c80-b124-7a84a7e92543\") " pod="openshift-ingress-canary/ingress-canary-klkjg"
Jul 08 16:34:55.637628 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.637609    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-klkjg"
Jul 08 16:34:55.761389 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.761120    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-klkjg"]
Jul 08 16:34:55.764001 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:55.763977    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0dfd2e0_9a06_4c80_b124_7a84a7e92543.slice/crio-2f3382233357236a3f3b334a142191c1f9b4944a0d2be17a2a73c8237e67475d WatchSource:0}: Error finding container 2f3382233357236a3f3b334a142191c1f9b4944a0d2be17a2a73c8237e67475d: Status 404 returned error can't find the container with id 2f3382233357236a3f3b334a142191c1f9b4944a0d2be17a2a73c8237e67475d
Jul 08 16:34:55.882481 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.882447    2575 generic.go:356] "Generic (PLEG): container finished" podID="2840b062-0201-4147-bc5e-17ceb2652dc7" containerID="c959c57061964e88a7be87dee3c5f83527c4f6e6c7ff831034cc09f2b3cb9b99" exitCode=0
Jul 08 16:34:55.882566 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.882532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bx962" event={"ID":"2840b062-0201-4147-bc5e-17ceb2652dc7","Type":"ContainerDied","Data":"c959c57061964e88a7be87dee3c5f83527c4f6e6c7ff831034cc09f2b3cb9b99"}
Jul 08 16:34:55.883688 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:55.883670    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-klkjg" event={"ID":"f0dfd2e0-9a06-4c80-b124-7a84a7e92543","Type":"ContainerStarted","Data":"2f3382233357236a3f3b334a142191c1f9b4944a0d2be17a2a73c8237e67475d"}
Jul 08 16:34:56.011227 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.011201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:56.011336 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.011307    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:56.011380 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.011354    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls podName:fa748963-12ef-4075-8741-ab58379b8a60 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.011340144 +0000 UTC m=+38.000896865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls") pod "dns-default-p8lx6" (UID: "fa748963-12ef-4075-8741-ab58379b8a60") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:56.603399 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.603364    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:34:56.603582 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.603400    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:34:56.603582 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.603406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:34:56.606166 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.606133    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.607229 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.607210    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j7mv4\""
Jul 08 16:34:56.607449 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.607406    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:34:56.608719 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.608499    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krkqn\""
Jul 08 16:34:56.608719 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.608526    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:34:56.608719 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.608585    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.676379 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.676360    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"]
Jul 08 16:34:56.678370 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.678353    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.680893 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.680863    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.681156 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.681138    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.681233 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.681197    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 08 16:34:56.681233 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.681214    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 08 16:34:56.681334 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.681199    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nrqkl\""
Jul 08 16:34:56.688396 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.688376    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"]
Jul 08 16:34:56.714785 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.714760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8319143d-9ddf-4074-b9ef-1acaffa068f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.714881 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.714830    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffbb\" (UniqueName: \"kubernetes.io/projected/8319143d-9ddf-4074-b9ef-1acaffa068f0-kube-api-access-5ffbb\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.714931 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.714890    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8319143d-9ddf-4074-b9ef-1acaffa068f0-config\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.801887 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.801849    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"]
Jul 08 16:34:56.803709 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.803688    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f9489547d-l49zd"]
Jul 08 16:34:56.803831 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.803767    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.805508 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.805489    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"]
Jul 08 16:34:56.805623 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.805597    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.806289 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.806266    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.806381 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.806334    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.806610 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.806592    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jrh7q\""
Jul 08 16:34:56.807542 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.807467    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:34:56.807711 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.807681    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 08 16:34:56.808383 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.808270    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 16:34:56.808383 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.808286    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 16:34:56.808383 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.808308    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 16:34:56.808383 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.808322    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 08 16:34:56.808621 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.808443    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tqktz\""
Jul 08 16:34:56.810868 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.810849    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bplpp\""
Jul 08 16:34:56.810969 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.810893    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 08 16:34:56.811037 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.810855    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.811037 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.811009    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.815577 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7xt5\" (UniqueName: \"kubernetes.io/projected/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-kube-api-access-x7xt5\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.815577 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxfz\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-kube-api-access-tjxfz\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.815769 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-image-registry-private-configuration\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.815769 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffbb\" (UniqueName: \"kubernetes.io/projected/8319143d-9ddf-4074-b9ef-1acaffa068f0-kube-api-access-5ffbb\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.815769 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815693    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-ca-trust-extracted\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.815769 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:34:56.815769 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8319143d-9ddf-4074-b9ef-1acaffa068f0-config\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.816012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815820    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-installation-pull-secrets\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.816012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815853    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.816012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815877    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.816012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-bound-sa-token\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.816012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.816012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.815975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dtn\" (UniqueName: \"kubernetes.io/projected/a3b0727d-a655-4973-b190-6c949f6ef318-kube-api-access-q2dtn\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:34:56.816378 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.816194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8319143d-9ddf-4074-b9ef-1acaffa068f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.816378 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.816232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-certificates\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.816378 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.816274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-trusted-ca\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.816622 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.816582    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8319143d-9ddf-4074-b9ef-1acaffa068f0-config\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.820780 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.819979    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"]
Jul 08 16:34:56.820780 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.820690    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8319143d-9ddf-4074-b9ef-1acaffa068f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.821356 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.821316    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"]
Jul 08 16:34:56.822102 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.822082    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 16:34:56.822238 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.822215    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f9489547d-l49zd"]
Jul 08 16:34:56.848405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.848386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffbb\" (UniqueName: \"kubernetes.io/projected/8319143d-9ddf-4074-b9ef-1acaffa068f0-kube-api-access-5ffbb\") pod \"service-ca-operator-5d46d9ffd4-tmb45\" (UID: \"8319143d-9ddf-4074-b9ef-1acaffa068f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.889854 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.889799    2575 generic.go:356] "Generic (PLEG): container finished" podID="2840b062-0201-4147-bc5e-17ceb2652dc7" containerID="b354e9aed5fd87eda0b7aca717c65ef72ff5752220d99e7fab37bdda87b37227" exitCode=0
Jul 08 16:34:56.889854 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.889836    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bx962" event={"ID":"2840b062-0201-4147-bc5e-17ceb2652dc7","Type":"ContainerDied","Data":"b354e9aed5fd87eda0b7aca717c65ef72ff5752220d99e7fab37bdda87b37227"}
Jul 08 16:34:56.917198 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-ca-trust-extracted\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.917198 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:34:56.917349 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-installation-pull-secrets\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.917349 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917276    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.917349 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.917349 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-bound-sa-token\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.917547 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.917350    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:34:56.917547 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.917409    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls podName:a3b0727d-a655-4973-b190-6c949f6ef318 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.417390558 +0000 UTC m=+38.406947286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-z7rsl" (UID: "a3b0727d-a655-4973-b190-6c949f6ef318") : secret "samples-operator-tls" not found
Jul 08 16:34:56.917547 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.917547 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.917426    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:56.917547 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dtn\" (UniqueName: \"kubernetes.io/projected/a3b0727d-a655-4973-b190-6c949f6ef318-kube-api-access-q2dtn\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:34:56.917547 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-certificates\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.917547 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-trusted-ca\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.917916 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917548    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-ca-trust-extracted\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.917916 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.917588    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls podName:afcc2e47-6a0a-498e-9e3d-3ab061a32e71 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.417577236 +0000 UTC m=+38.407133966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gd5xv" (UID: "afcc2e47-6a0a-498e-9e3d-3ab061a32e71") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:56.917916 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917702    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7xt5\" (UniqueName: \"kubernetes.io/projected/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-kube-api-access-x7xt5\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.917916 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxfz\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-kube-api-access-tjxfz\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.917916 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.917766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-image-registry-private-configuration\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.918390 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.918368    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-certificates\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.919285 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.918807    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:56.919285 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.918825    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9489547d-l49zd: secret "image-registry-tls" not found
Jul 08 16:34:56.919285 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:56.918871    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls podName:c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.418852853 +0000 UTC m=+38.408409575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls") pod "image-registry-f9489547d-l49zd" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40") : secret "image-registry-tls" not found
Jul 08 16:34:56.919285 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.919243    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.919285 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.919271    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-trusted-ca\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.920624 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.920473    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f8778bf4f-tlgl8"]
Jul 08 16:34:56.922409 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.922385    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-installation-pull-secrets\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.923233 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.923199    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-image-registry-private-configuration\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.923985 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.923967    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:56.926974 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.926953    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.927677 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.927170    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 08 16:34:56.927677 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.927362    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 08 16:34:56.927677 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.927616    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l74bm\""
Jul 08 16:34:56.928269 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.927971    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.928269 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.928082    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 08 16:34:56.928269 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.928173    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 08 16:34:56.946964 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.945386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dtn\" (UniqueName: \"kubernetes.io/projected/a3b0727d-a655-4973-b190-6c949f6ef318-kube-api-access-q2dtn\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:34:56.946964 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.946740    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7xt5\" (UniqueName: \"kubernetes.io/projected/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-kube-api-access-x7xt5\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:56.946964 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.946826    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f8778bf4f-tlgl8"]
Jul 08 16:34:56.947156 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.947123    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-bound-sa-token\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.947995 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.947779    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxfz\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-kube-api-access-tjxfz\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:56.987335 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.987201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"
Jul 08 16:34:56.998767 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:56.998738    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-mdgr8"]
Jul 08 16:34:57.001161 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.001140    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.005163 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.004700    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 16:34:57.005163 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.004775    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vmmfv\""
Jul 08 16:34:57.005163 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.004915    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 08 16:34:57.005163 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.004972    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 08 16:34:57.005405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.005179    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 16:34:57.009958 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.009880    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 08 16:34:57.012500 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.012479    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-mdgr8"]
Jul 08 16:34:57.018605 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.018584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/650f88a0-c7d3-44d7-b457-d81d1584808c-tmp\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.018722 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.018628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/650f88a0-c7d3-44d7-b457-d81d1584808c-serving-cert\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.018722 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.018680    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-stats-auth\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.018836 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.018808    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.018893 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.018844    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/650f88a0-c7d3-44d7-b457-d81d1584808c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.018991 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.018965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/650f88a0-c7d3-44d7-b457-d81d1584808c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.019097 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.018997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/650f88a0-c7d3-44d7-b457-d81d1584808c-snapshots\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.019157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.019090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.019157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.019136    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:57.019262 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.019159    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc27r\" (UniqueName: \"kubernetes.io/projected/650f88a0-c7d3-44d7-b457-d81d1584808c-kube-api-access-tc27r\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.019315 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.019256    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:57.019315 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.019313    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls podName:fa748963-12ef-4075-8741-ab58379b8a60 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:59.019293943 +0000 UTC m=+40.008850687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls") pod "dns-default-p8lx6" (UID: "fa748963-12ef-4075-8741-ab58379b8a60") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:57.019421 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.019361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctgg\" (UniqueName: \"kubernetes.io/projected/9edb90cb-255d-4637-9bd9-14e8c87c3c06-kube-api-access-bctgg\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.019421 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.019391    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-default-certificate\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.119927 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.119895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/650f88a0-c7d3-44d7-b457-d81d1584808c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.119927 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.119933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/650f88a0-c7d3-44d7-b457-d81d1584808c-snapshots\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.120139 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.119984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.120139 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc27r\" (UniqueName: \"kubernetes.io/projected/650f88a0-c7d3-44d7-b457-d81d1584808c-kube-api-access-tc27r\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.120139 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctgg\" (UniqueName: \"kubernetes.io/projected/9edb90cb-255d-4637-9bd9-14e8c87c3c06-kube-api-access-bctgg\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.120139 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120110    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-default-certificate\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.120321 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/650f88a0-c7d3-44d7-b457-d81d1584808c-tmp\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.120321 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/650f88a0-c7d3-44d7-b457-d81d1584808c-serving-cert\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.120321 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-stats-auth\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.120593 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120570    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/650f88a0-c7d3-44d7-b457-d81d1584808c-snapshots\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.120675 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.120595    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.620572963 +0000 UTC m=+38.610129691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : configmap references non-existent config key: service-ca.crt
Jul 08 16:34:57.120675 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.120793 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.120696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/650f88a0-c7d3-44d7-b457-d81d1584808c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.121109 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.121040    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:34:57.121109 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.121074    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/650f88a0-c7d3-44d7-b457-d81d1584808c-tmp\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.121268 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.121117    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.621082509 +0000 UTC m=+38.610639243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : secret "router-metrics-certs-default" not found
Jul 08 16:34:57.122103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.122059    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/650f88a0-c7d3-44d7-b457-d81d1584808c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.122920 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.122871    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/650f88a0-c7d3-44d7-b457-d81d1584808c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.123791 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.123761    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/650f88a0-c7d3-44d7-b457-d81d1584808c-serving-cert\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.123974 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.123953    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45"]
Jul 08 16:34:57.124033 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.124020    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-default-certificate\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.124426 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.124407    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-stats-auth\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.128802 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:57.128774    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8319143d_9ddf_4074_b9ef_1acaffa068f0.slice/crio-2667b648749c0cab72d391891beeed8c1b98ec6a0f12acace582d09a0f75c627 WatchSource:0}: Error finding container 2667b648749c0cab72d391891beeed8c1b98ec6a0f12acace582d09a0f75c627: Status 404 returned error can't find the container with id 2667b648749c0cab72d391891beeed8c1b98ec6a0f12acace582d09a0f75c627
Jul 08 16:34:57.129304 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.129280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:57.129423 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.129300    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctgg\" (UniqueName: \"kubernetes.io/projected/9edb90cb-255d-4637-9bd9-14e8c87c3c06-kube-api-access-bctgg\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.129423 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.129386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc27r\" (UniqueName: \"kubernetes.io/projected/650f88a0-c7d3-44d7-b457-d81d1584808c-kube-api-access-tc27r\") pod \"insights-operator-5c847dcfc4-mdgr8\" (UID: \"650f88a0-c7d3-44d7-b457-d81d1584808c\") " pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.246213 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.246131    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:57.314590 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.314564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8"
Jul 08 16:34:57.423488 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.423449    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:57.423664 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.423564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:34:57.423664 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.423598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:57.423664 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.423616    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:57.423808 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.423706    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls podName:afcc2e47-6a0a-498e-9e3d-3ab061a32e71 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.423686275 +0000 UTC m=+39.413242998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gd5xv" (UID: "afcc2e47-6a0a-498e-9e3d-3ab061a32e71") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:57.423808 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.423722    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:34:57.423808 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.423726    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:57.423808 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.423780    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls podName:a3b0727d-a655-4973-b190-6c949f6ef318 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.423762166 +0000 UTC m=+39.413318902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-z7rsl" (UID: "a3b0727d-a655-4973-b190-6c949f6ef318") : secret "samples-operator-tls" not found
Jul 08 16:34:57.423808 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.423782    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9489547d-l49zd: secret "image-registry-tls" not found
Jul 08 16:34:57.424023 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.423846    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls podName:c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.423827854 +0000 UTC m=+39.413384592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls") pod "image-registry-f9489547d-l49zd" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40") : secret "image-registry-tls" not found
Jul 08 16:34:57.625213 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.625180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.625380 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.625298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:57.625380 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.625329    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:34:57.625496 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.625393    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.625375625 +0000 UTC m=+39.614932366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : secret "router-metrics-certs-default" not found
Jul 08 16:34:57.625561 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:57.625500    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.625481684 +0000 UTC m=+39.615038410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : configmap references non-existent config key: service-ca.crt
Jul 08 16:34:57.894314 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.894283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45" event={"ID":"8319143d-9ddf-4074-b9ef-1acaffa068f0","Type":"ContainerStarted","Data":"2667b648749c0cab72d391891beeed8c1b98ec6a0f12acace582d09a0f75c627"}
Jul 08 16:34:57.897637 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.897614    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bx962" event={"ID":"2840b062-0201-4147-bc5e-17ceb2652dc7","Type":"ContainerStarted","Data":"89097bdc74845330b386efec4d6706f17fb59436fba4333b95b5b98bf656184c"}
Jul 08 16:34:57.921854 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.921812    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bx962" podStartSLOduration=9.515073336 podStartE2EDuration="38.921798203s" totalImagesPullingTime="29.406724867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:57.920200338 +0000 UTC m=+38.909757103" watchObservedRunningTime="2026-07-08 16:34:57.921798203 +0000 UTC m=+38.911354946"
Jul 08 16:34:57.956925 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:57.956721    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-mdgr8"]
Jul 08 16:34:57.959635 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:34:57.959603    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650f88a0_c7d3_44d7_b457_d81d1584808c.slice/crio-ce3840610d8e088fe274b27383998ff74da8975c504878926434e63da3979c99 WatchSource:0}: Error finding container ce3840610d8e088fe274b27383998ff74da8975c504878926434e63da3979c99: Status 404 returned error can't find the container with id ce3840610d8e088fe274b27383998ff74da8975c504878926434e63da3979c99
Jul 08 16:34:58.053119 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.053077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:58.197761 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.197628    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:58.301400 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.301327    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:58.431324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.431292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:34:58.431484 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.431389    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:34:58.431484 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.431462    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:58.431484 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.431479    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9489547d-l49zd: secret "image-registry-tls" not found
Jul 08 16:34:58.431640 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.431483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:34:58.431640 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.431537    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls podName:c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.431517097 +0000 UTC m=+41.421073841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls") pod "image-registry-f9489547d-l49zd" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40") : secret "image-registry-tls" not found
Jul 08 16:34:58.431640 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.431579    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:34:58.431640 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.431615    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:58.431936 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.431623    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls podName:a3b0727d-a655-4973-b190-6c949f6ef318 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.431611652 +0000 UTC m=+41.421168378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-z7rsl" (UID: "a3b0727d-a655-4973-b190-6c949f6ef318") : secret "samples-operator-tls" not found
Jul 08 16:34:58.431936 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.431710    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls podName:afcc2e47-6a0a-498e-9e3d-3ab061a32e71 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.431687446 +0000 UTC m=+41.421244176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gd5xv" (UID: "afcc2e47-6a0a-498e-9e3d-3ab061a32e71") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:58.501150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.501077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:58.602989 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.602934    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:58.633475 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.633450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:58.633641 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.633536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:34:58.633641 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.633616    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:34:58.633768 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.633690    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.633675368 +0000 UTC m=+41.623232094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : secret "router-metrics-certs-default" not found
Jul 08 16:34:58.633768 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:58.633709    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.633700768 +0000 UTC m=+41.623257491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : configmap references non-existent config key: service-ca.crt
Jul 08 16:34:58.901199 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.901166    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-klkjg" event={"ID":"f0dfd2e0-9a06-4c80-b124-7a84a7e92543","Type":"ContainerStarted","Data":"159af6e30fc0f571789073c56c7416aa0a5e36dacb0dc245bd7a64d388915c3b"}
Jul 08 16:34:58.902457 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.902432    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8" event={"ID":"650f88a0-c7d3-44d7-b457-d81d1584808c","Type":"ContainerStarted","Data":"ce3840610d8e088fe274b27383998ff74da8975c504878926434e63da3979c99"}
Jul 08 16:34:58.918461 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:58.918418    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-klkjg" podStartSLOduration=1.816464181 podStartE2EDuration="3.91840162s" totalImagesPullingTime="2.101937439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:58.918029541 +0000 UTC m=+39.907586295" watchObservedRunningTime="2026-07-08 16:34:58.91840162 +0000 UTC m=+39.907958341"
Jul 08 16:34:59.038754 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:59.038723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:34:59.038916 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:59.038888    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:59.038974 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:34:59.038965    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls podName:fa748963-12ef-4075-8741-ab58379b8a60 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:03.038944712 +0000 UTC m=+44.028501448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls") pod "dns-default-p8lx6" (UID: "fa748963-12ef-4075-8741-ab58379b8a60") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:59.086454 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:59.086424    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:59.173955 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:59.173808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:59.398195 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:59.398164    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:59.500246 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:34:59.500120    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:00.453400 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.453371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.453439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.453465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.453537    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.453603    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls podName:a3b0727d-a655-4973-b190-6c949f6ef318 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.453587228 +0000 UTC m=+45.443143953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-z7rsl" (UID: "a3b0727d-a655-4973-b190-6c949f6ef318") : secret "samples-operator-tls" not found
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.453537    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.453687    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls podName:afcc2e47-6a0a-498e-9e3d-3ab061a32e71 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.453675349 +0000 UTC m=+45.443232069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gd5xv" (UID: "afcc2e47-6a0a-498e-9e3d-3ab061a32e71") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.453545    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.453704    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9489547d-l49zd: secret "image-registry-tls" not found
Jul 08 16:35:00.453843 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.453725    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls podName:c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.453718074 +0000 UTC m=+45.443274815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls") pod "image-registry-f9489547d-l49zd" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40") : secret "image-registry-tls" not found
Jul 08 16:35:00.654716 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.654692    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:00.654815 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.654798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:00.654882 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.654829    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.654816425 +0000 UTC m=+45.644373146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : configmap references non-existent config key: service-ca.crt
Jul 08 16:35:00.654941 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.654901    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:35:00.654990 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:00.654953    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.65493859 +0000 UTC m=+45.644495313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : secret "router-metrics-certs-default" not found
Jul 08 16:35:00.909354 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.909319    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45" event={"ID":"8319143d-9ddf-4074-b9ef-1acaffa068f0","Type":"ContainerStarted","Data":"d4c141fbed7d1ee000aa1b309cd4b055ebb82ff8281b1da5d9d8ef0025fa4a9b"}
Jul 08 16:35:00.910861 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.910832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8" event={"ID":"650f88a0-c7d3-44d7-b457-d81d1584808c","Type":"ContainerStarted","Data":"f3eecc8cd24338c1772a6cbb1049505e5e13c35c5dc5f74b73b31ba4e1c337e5"}
Jul 08 16:35:00.925774 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.925726    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45" podStartSLOduration=1.419380752 podStartE2EDuration="4.925706861s" totalImagesPullingTime="3.506326109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:00.924242905 +0000 UTC m=+41.913799648" watchObservedRunningTime="2026-07-08 16:35:00.925706861 +0000 UTC m=+41.915263605"
Jul 08 16:35:00.942130 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:00.942027    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8" podStartSLOduration=2.262376813 podStartE2EDuration="4.942015149s" totalImagesPullingTime="2.679638336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:00.941627035 +0000 UTC m=+41.931183791" watchObservedRunningTime="2026-07-08 16:35:00.942015149 +0000 UTC m=+41.931571893"
Jul 08 16:35:01.017594 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:01.017533    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:01.316208 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:01.316097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:01.410490 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:01.410458    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:01.801151 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:01.801123    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:02.017303 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.017273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:02.234748 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.234716    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:02.369053 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.369015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:02.372116 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.372094    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls"]
Jul 08 16:35:02.402892 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.402863    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-psrtc"]
Jul 08 16:35:02.403019 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.402975    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls"
Jul 08 16:35:02.406879 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.406858    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kwxvz\""
Jul 08 16:35:02.407040 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.407013    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 08 16:35:02.407903 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.407879    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 08 16:35:02.416775 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.416759    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls"]
Jul 08 16:35:02.416853 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.416789    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-psrtc"]
Jul 08 16:35:02.416887 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.416872    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.419908 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.419883    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 16:35:02.420007 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.419909    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 16:35:02.420055 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.420008    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5r9vg\""
Jul 08 16:35:02.463628 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.463602    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:02.469565 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.469543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-data-volume\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.469668 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.469578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fftw\" (UniqueName: \"kubernetes.io/projected/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-api-access-6fftw\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.469739 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.469690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.469792 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.469740    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.469792 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.469773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-crio-socket\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.469887 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.469803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6zww\" (UniqueName: \"kubernetes.io/projected/0a874a90-715a-43b8-9e06-99ade98392b5-kube-api-access-b6zww\") pod \"migrator-d57d58cc9-hgjls\" (UID: \"0a874a90-715a-43b8-9e06-99ade98392b5\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls"
Jul 08 16:35:02.570621 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.570567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.570621 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.570602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.570621 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.570622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-crio-socket\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.570823 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:02.570689    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:02.570823 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:02.570717    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:02.570823 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:02.570731    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:03.070717793 +0000 UTC m=+44.060274518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:02.570823 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.570766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6zww\" (UniqueName: \"kubernetes.io/projected/0a874a90-715a-43b8-9e06-99ade98392b5-kube-api-access-b6zww\") pod \"migrator-d57d58cc9-hgjls\" (UID: \"0a874a90-715a-43b8-9e06-99ade98392b5\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls"
Jul 08 16:35:02.570823 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:02.570776    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:03.070761229 +0000 UTC m=+44.060317950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:02.571062 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.570887    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-crio-socket\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.571062 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.570923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-data-volume\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.571062 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.570955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fftw\" (UniqueName: \"kubernetes.io/projected/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-api-access-6fftw\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.571243 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.571202    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-data-volume\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.580145 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.580126    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6zww\" (UniqueName: \"kubernetes.io/projected/0a874a90-715a-43b8-9e06-99ade98392b5-kube-api-access-b6zww\") pod \"migrator-d57d58cc9-hgjls\" (UID: \"0a874a90-715a-43b8-9e06-99ade98392b5\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls"
Jul 08 16:35:02.580230 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.580165    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fftw\" (UniqueName: \"kubernetes.io/projected/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-api-access-6fftw\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:02.622441 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.622418    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:02.712910 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.712883    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls"
Jul 08 16:35:02.851503 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.851478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls"]
Jul 08 16:35:02.853629 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:02.853602    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a874a90_715a_43b8_9e06_99ade98392b5.slice/crio-fafc3e19ac1c38f8ea55fefb6c879c1df364bcb553b171dac9673a05cb3dd758 WatchSource:0}: Error finding container fafc3e19ac1c38f8ea55fefb6c879c1df364bcb553b171dac9673a05cb3dd758: Status 404 returned error can't find the container with id fafc3e19ac1c38f8ea55fefb6c879c1df364bcb553b171dac9673a05cb3dd758
Jul 08 16:35:02.916893 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:02.916868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls" event={"ID":"0a874a90-715a-43b8-9e06-99ade98392b5","Type":"ContainerStarted","Data":"fafc3e19ac1c38f8ea55fefb6c879c1df364bcb553b171dac9673a05cb3dd758"}
Jul 08 16:35:03.075188 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:03.075161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:03.075278 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:03.075195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:35:03.075278 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:03.075213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:03.075390 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:03.075299    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:03.075390 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:03.075322    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:03.075390 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:03.075358    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.075344049 +0000 UTC m=+45.064900774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:03.075390 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:03.075384    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls podName:fa748963-12ef-4075-8741-ab58379b8a60 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:11.075365012 +0000 UTC m=+52.064921737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls") pod "dns-default-p8lx6" (UID: "fa748963-12ef-4075-8741-ab58379b8a60") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:03.075390 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:03.075327    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:03.075567 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:03.075427    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.07541644 +0000 UTC m=+45.064973161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:04.084068 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.083989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:04.084068 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.084032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:04.084490 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.084116    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:04.084490 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.084145    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:04.084490 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.084165    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:06.084152395 +0000 UTC m=+47.073709120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:04.084490 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.084193    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:06.084181023 +0000 UTC m=+47.073737749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:04.118942 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.118920    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vtcrd_370616ca-420a-4679-91d6-9c035007c735/dns-node-resolver/0.log"
Jul 08 16:35:04.151177 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.151154    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.255670 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.255624    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.487835 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.487806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:35:04.487992 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.487952    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:04.487992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.487984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:35:04.488107 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.488019    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls podName:afcc2e47-6a0a-498e-9e3d-3ab061a32e71 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.487999746 +0000 UTC m=+53.477556467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gd5xv" (UID: "afcc2e47-6a0a-498e-9e3d-3ab061a32e71") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:04.488107 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.488056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:04.488107 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.488061    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:04.488107 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.488104    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls podName:a3b0727d-a655-4973-b190-6c949f6ef318 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.488092965 +0000 UTC m=+53.477649686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-z7rsl" (UID: "a3b0727d-a655-4973-b190-6c949f6ef318") : secret "samples-operator-tls" not found
Jul 08 16:35:04.488310 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.488133    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:04.488310 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.488145    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f9489547d-l49zd: secret "image-registry-tls" not found
Jul 08 16:35:04.488310 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.488185    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls podName:c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.488174928 +0000 UTC m=+53.477731657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls") pod "image-registry-f9489547d-l49zd" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40") : secret "image-registry-tls" not found
Jul 08 16:35:04.689283 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.689252    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:04.689440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.689321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:04.689440 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.689418    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:35:04.689547 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.689424    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.689411268 +0000 UTC m=+53.678967989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : configmap references non-existent config key: service-ca.crt
Jul 08 16:35:04.689547 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:04.689488    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.689471251 +0000 UTC m=+53.679027987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : secret "router-metrics-certs-default" not found
Jul 08 16:35:04.916765 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:04.916739    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8p82f_e60e3631-e481-4969-9b02-2db258f40abc/node-ca/0.log"
Jul 08 16:35:05.190782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:05.190719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:05.517162 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:05.517097    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-klkjg_f0dfd2e0-9a06-4c80-b124-7a84a7e92543/serve-healthcheck-canary/0.log"
Jul 08 16:35:05.925831 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:05.925804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls" event={"ID":"0a874a90-715a-43b8-9e06-99ade98392b5","Type":"ContainerStarted","Data":"5f01ea452a11061c1f8418bfc4b3476d109298f196fd92a9b4961c035a49231f"}
Jul 08 16:35:05.926037 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:05.925858    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls" event={"ID":"0a874a90-715a-43b8-9e06-99ade98392b5","Type":"ContainerStarted","Data":"333cd8475e74337fe3bb1aaecd6f42c0b317b46b0f08f1281e88f2d51e6a56c7"}
Jul 08 16:35:05.946499 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:05.946448    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hgjls" podStartSLOduration=1.8589731440000001 podStartE2EDuration="3.94643214s" totalImagesPullingTime="2.087458996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:05.945295428 +0000 UTC m=+46.934852182" watchObservedRunningTime="2026-07-08 16:35:05.94643214 +0000 UTC m=+46.935988926"
Jul 08 16:35:06.101451 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:06.101422    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:06.101603 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:06.101461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:06.101603 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:06.101579    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:06.101603 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:06.101587    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:06.101737 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:06.101630    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:10.10161624 +0000 UTC m=+51.091172964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:06.101737 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:06.101665    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:10.101637649 +0000 UTC m=+51.091194370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:07.083070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:07.083038    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:07.251016 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:07.250987    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:07.325618 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:07.325570    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:07.426954 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:07.426927    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:07.804032 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:07.803969    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:07.959623 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:07.959593    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:08.759349 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:08.759318    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:08.947264 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:08.947233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:09.366428 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:09.366384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:09.556327 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:09.556293    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:09.955298 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:09.955269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.132799 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:10.132774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:10.132935 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:10.132810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:10.132935 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:10.132890    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:10.132935 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:10.132936    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:18.132923711 +0000 UTC m=+59.122480432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:10.133092 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:10.132932    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:10.133092 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:10.133010    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls podName:9099ba48-3525-45b0-a418-c0f0e9d4ba6f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:18.132990535 +0000 UTC m=+59.122547270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psrtc" (UID: "9099ba48-3525-45b0-a418-c0f0e9d4ba6f") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:10.670936 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:10.670904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.772195 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:10.772168    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.871353 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:10.871331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.973025 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:10.972961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:11.140150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:11.140127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:35:11.142337 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:11.142321    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa748963-12ef-4075-8741-ab58379b8a60-metrics-tls\") pod \"dns-default-p8lx6\" (UID: \"fa748963-12ef-4075-8741-ab58379b8a60\") " pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:35:11.158543 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:11.158521    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:11.229306 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:11.229261    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:35:11.369115 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:11.369099    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p8lx6"]
Jul 08 16:35:11.371596 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:11.371575    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa748963_12ef_4075_8741_ab58379b8a60.slice/crio-4ae70ecae8e9904aac4108830f3d3754c9aa048b966a4e280d36fc7d4f0a0fb4 WatchSource:0}: Error finding container 4ae70ecae8e9904aac4108830f3d3754c9aa048b966a4e280d36fc7d4f0a0fb4: Status 404 returned error can't find the container with id 4ae70ecae8e9904aac4108830f3d3754c9aa048b966a4e280d36fc7d4f0a0fb4
Jul 08 16:35:11.568005 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:11.567947    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:11.628868 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:11.628842    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:11.944292 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:11.944270    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8lx6" event={"ID":"fa748963-12ef-4075-8741-ab58379b8a60","Type":"ContainerStarted","Data":"4ae70ecae8e9904aac4108830f3d3754c9aa048b966a4e280d36fc7d4f0a0fb4"}
Jul 08 16:35:12.382686 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.382633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:12.551392 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.551357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:35:12.551543 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.551490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:12.551603 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.551563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:35:12.551748 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:12.551727    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:12.551825 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:12.551810    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls podName:afcc2e47-6a0a-498e-9e3d-3ab061a32e71 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:28.551786031 +0000 UTC m=+69.541342754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gd5xv" (UID: "afcc2e47-6a0a-498e-9e3d-3ab061a32e71") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:12.554103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.554074    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b0727d-a655-4973-b190-6c949f6ef318-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-z7rsl\" (UID: \"a3b0727d-a655-4973-b190-6c949f6ef318\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:35:12.554274 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.554256    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"image-registry-f9489547d-l49zd\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") " pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:12.588480 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.588446    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:12.733022 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.732956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:12.737050 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.737029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"
Jul 08 16:35:12.753441 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.753417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:12.753538 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.753527    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:12.753627 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:12.753604    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle podName:9edb90cb-255d-4637-9bd9-14e8c87c3c06 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:28.753582959 +0000 UTC m=+69.743139696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle") pod "router-default-7f8778bf4f-tlgl8" (UID: "9edb90cb-255d-4637-9bd9-14e8c87c3c06") : configmap references non-existent config key: service-ca.crt
Jul 08 16:35:12.756272 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.756244    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edb90cb-255d-4637-9bd9-14e8c87c3c06-metrics-certs\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:12.896300 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.896268    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f9489547d-l49zd"]
Jul 08 16:35:12.899612 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:12.899588    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a38bfc_b69f_48f0_a6c0_e6a793ad9c40.slice/crio-0481526e148ffbd8058235c22b5f75dff8b6d9308d6dccc8a43ffec98c3b9218 WatchSource:0}: Error finding container 0481526e148ffbd8058235c22b5f75dff8b6d9308d6dccc8a43ffec98c3b9218: Status 404 returned error can't find the container with id 0481526e148ffbd8058235c22b5f75dff8b6d9308d6dccc8a43ffec98c3b9218
Jul 08 16:35:12.912452 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.912433    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl"]
Jul 08 16:35:12.949200 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.949169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f9489547d-l49zd" event={"ID":"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40","Type":"ContainerStarted","Data":"0481526e148ffbd8058235c22b5f75dff8b6d9308d6dccc8a43ffec98c3b9218"}
Jul 08 16:35:12.958461 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:12.958439    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.149945 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.149891    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.436835 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.436789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.584323 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.584287    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.872843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.872814    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.953756 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.953728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl" event={"ID":"a3b0727d-a655-4973-b190-6c949f6ef318","Type":"ContainerStarted","Data":"184875b0cb238005a270daa5156349f9bfc264b2ee90f66300d430ad79eacbbe"}
Jul 08 16:35:13.955551 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.955526    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f9489547d-l49zd" event={"ID":"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40","Type":"ContainerStarted","Data":"d759eedffedbe3a49659c5f2a5a73d3db185c4be0bf965c4b3049a58b634ae24"}
Jul 08 16:35:13.955698 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.955681    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:13.971318 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.971294    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.977724 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:13.977669    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f9489547d-l49zd" podStartSLOduration=17.9776368 podStartE2EDuration="17.9776368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:13.975512864 +0000 UTC m=+54.965069656" watchObservedRunningTime="2026-07-08 16:35:13.9776368 +0000 UTC m=+54.967193544"
Jul 08 16:35:14.157872 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:14.157789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.466010 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:14.465928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.868062 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:14.868025    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.961035 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:14.961006    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8lx6" event={"ID":"fa748963-12ef-4075-8741-ab58379b8a60","Type":"ContainerStarted","Data":"484fd6dc83cdb136951a5df80cca09a98e1da52c3cf6fb998c80b45cf49193e7"}
Jul 08 16:35:14.961200 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:14.961061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8lx6" event={"ID":"fa748963-12ef-4075-8741-ab58379b8a60","Type":"ContainerStarted","Data":"4e60a11deb029946e577cfa5c5f940760727284de543120b9cbd99921e5232a8"}
Jul 08 16:35:14.966114 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:14.966083    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.988479 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:14.988426    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p8lx6" podStartSLOduration=17.404739467 podStartE2EDuration="19.988407143s" totalImagesPullingTime="2.583667676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:14.986174231 +0000 UTC m=+55.975730966" watchObservedRunningTime="2026-07-08 16:35:14.988407143 +0000 UTC m=+55.977963887"
Jul 08 16:35:15.395633 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:15.395591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.573746 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:15.573708    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.965731 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:15.965667    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl" event={"ID":"a3b0727d-a655-4973-b190-6c949f6ef318","Type":"ContainerStarted","Data":"8739b39b18f25ec221ea75ba651f09e9b668db876ef51d44eecf1fd690997cb1"}
Jul 08 16:35:15.965731 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:15.965720    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl" event={"ID":"a3b0727d-a655-4973-b190-6c949f6ef318","Type":"ContainerStarted","Data":"4cd78321709ea366dbd315b1bb38df6d08df76cef2cdb64536df6508f247ac67"}
Jul 08 16:35:15.965955 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:15.965941    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:35:16.178012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:16.177982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:16.762201 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:16.762167    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:16.880736 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:16.880709    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.071443 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:17.071372    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.179025 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:17.179005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.362167 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:17.362142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.753737 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:17.753676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:18.199729 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:18.199697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:18.200023 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:18.199738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:18.200397 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:18.200372    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:18.202092 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:18.202075    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9099ba48-3525-45b0-a418-c0f0e9d4ba6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psrtc\" (UID: \"9099ba48-3525-45b0-a418-c0f0e9d4ba6f\") " pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:18.324769 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:18.324746    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-psrtc"
Jul 08 16:35:18.447263 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:18.447215    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-z7rsl" podStartSLOduration=19.791948695 podStartE2EDuration="22.44719649s" totalImagesPullingTime="2.655247795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:15.985296751 +0000 UTC m=+56.974853532" watchObservedRunningTime="2026-07-08 16:35:18.44719649 +0000 UTC m=+59.436753232"
Jul 08 16:35:18.448009 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:18.447990    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-psrtc"]
Jul 08 16:35:18.450179 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:18.450108    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9099ba48_3525_45b0_a418_c0f0e9d4ba6f.slice/crio-9b1a091b40c2ef61b6d34df8570155bfe8ec59c662ef77b0b6572a60f531c161 WatchSource:0}: Error finding container 9b1a091b40c2ef61b6d34df8570155bfe8ec59c662ef77b0b6572a60f531c161: Status 404 returned error can't find the container with id 9b1a091b40c2ef61b6d34df8570155bfe8ec59c662ef77b0b6572a60f531c161
Jul 08 16:35:18.975806 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:18.975766    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-psrtc" event={"ID":"9099ba48-3525-45b0-a418-c0f0e9d4ba6f","Type":"ContainerStarted","Data":"9b1a091b40c2ef61b6d34df8570155bfe8ec59c662ef77b0b6572a60f531c161"}
Jul 08 16:35:21.882893 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:21.882862    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pcqm"
Jul 08 16:35:22.936249 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:22.936227    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.991153 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:22.991122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-psrtc" event={"ID":"9099ba48-3525-45b0-a418-c0f0e9d4ba6f","Type":"ContainerStarted","Data":"555ad15d80af885b72ece5fe41d29fa3a11077deab812c814c7e34f056011b21"}
Jul 08 16:35:22.991251 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:22.991185    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-psrtc" event={"ID":"9099ba48-3525-45b0-a418-c0f0e9d4ba6f","Type":"ContainerStarted","Data":"affb8da6bdf7ba46cc74648791a66397fae94ca7d0a02ac1753e08548e53cf4b"}
Jul 08 16:35:23.040721 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:23.040690    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:25.261505 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.261466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:35:25.265121 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.265099    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:35:25.274056 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.274036    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a323b89-9740-4ab1-9d0d-1ca2bec3d93d-original-pull-secret\") pod \"global-pull-secret-syncer-v8x8z\" (UID: \"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d\") " pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:35:25.361950 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.361916    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:35:25.364497 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.364477    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:35:25.374112 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.374085    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8751bafd-f26c-47bd-8fa8-f24755c1fb18-metrics-certs\") pod \"network-metrics-daemon-4x4wh\" (UID: \"8751bafd-f26c-47bd-8fa8-f24755c1fb18\") " pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:35:25.421489 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.421467    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krkqn\""
Jul 08 16:35:25.428656 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.428635    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x4wh"
Jul 08 16:35:25.431546 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.431531    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8x8z"
Jul 08 16:35:25.462386 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.462354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:35:25.465225 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.465201    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:35:25.476174 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.475915    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:35:25.492286 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.491590    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkgc\" (UniqueName: \"kubernetes.io/projected/2f1fa945-7def-4e82-9da9-78010a748562-kube-api-access-tvkgc\") pod \"network-check-target-dtcdw\" (UID: \"2f1fa945-7def-4e82-9da9-78010a748562\") " pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:35:25.575748 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.575720    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4x4wh"]
Jul 08 16:35:25.577534 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:25.577511    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751bafd_f26c_47bd_8fa8_f24755c1fb18.slice/crio-650a4e7d232deada0be2e16e1067da96ae9fae43e755a2be48bc6c24e1aa7c09 WatchSource:0}: Error finding container 650a4e7d232deada0be2e16e1067da96ae9fae43e755a2be48bc6c24e1aa7c09: Status 404 returned error can't find the container with id 650a4e7d232deada0be2e16e1067da96ae9fae43e755a2be48bc6c24e1aa7c09
Jul 08 16:35:25.591917 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.591897    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v8x8z"]
Jul 08 16:35:25.593936 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:25.593915    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a323b89_9740_4ab1_9d0d_1ca2bec3d93d.slice/crio-e3cda070d925f2559e1c79dc0cdd4436520af1f69f08d2d5f136e595cc4548e3 WatchSource:0}: Error finding container e3cda070d925f2559e1c79dc0cdd4436520af1f69f08d2d5f136e595cc4548e3: Status 404 returned error can't find the container with id e3cda070d925f2559e1c79dc0cdd4436520af1f69f08d2d5f136e595cc4548e3
Jul 08 16:35:25.727770 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.727744    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j7mv4\""
Jul 08 16:35:25.735672 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.735637    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:35:25.851044 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.851000    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dtcdw"]
Jul 08 16:35:25.852981 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:25.852955    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1fa945_7def_4e82_9da9_78010a748562.slice/crio-cca82171f7957c3204c170f37cd3335273c0b41e2566efada349e387cf3b3a58 WatchSource:0}: Error finding container cca82171f7957c3204c170f37cd3335273c0b41e2566efada349e387cf3b3a58: Status 404 returned error can't find the container with id cca82171f7957c3204c170f37cd3335273c0b41e2566efada349e387cf3b3a58
Jul 08 16:35:25.969519 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:25.969499    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p8lx6"
Jul 08 16:35:26.001866 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:26.001836    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-psrtc" event={"ID":"9099ba48-3525-45b0-a418-c0f0e9d4ba6f","Type":"ContainerStarted","Data":"398b0918c0b6f56d59cdd3bfb949b779c852fe057fb6238a27426abf9e2dcaa4"}
Jul 08 16:35:26.002946 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:26.002927    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dtcdw" event={"ID":"2f1fa945-7def-4e82-9da9-78010a748562","Type":"ContainerStarted","Data":"cca82171f7957c3204c170f37cd3335273c0b41e2566efada349e387cf3b3a58"}
Jul 08 16:35:26.003926 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:26.003896    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v8x8z" event={"ID":"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d","Type":"ContainerStarted","Data":"e3cda070d925f2559e1c79dc0cdd4436520af1f69f08d2d5f136e595cc4548e3"}
Jul 08 16:35:26.004932 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:26.004912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4x4wh" event={"ID":"8751bafd-f26c-47bd-8fa8-f24755c1fb18","Type":"ContainerStarted","Data":"650a4e7d232deada0be2e16e1067da96ae9fae43e755a2be48bc6c24e1aa7c09"}
Jul 08 16:35:26.020069 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:26.020021    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-psrtc" podStartSLOduration=17.243950116 podStartE2EDuration="24.020003145s" totalImagesPullingTime="6.776053029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:35:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:26.019440718 +0000 UTC m=+67.008997489" watchObservedRunningTime="2026-07-08 16:35:26.020003145 +0000 UTC m=+67.009559889"
Jul 08 16:35:26.605361 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:26.605322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:26.706779 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:26.706743    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:26.810260 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:26.810218    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.162474 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.162447    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.556675 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.556608    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.623040 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.622902    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.694890 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.694856    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.875091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.875062    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"]
Jul 08 16:35:27.906907 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.906873    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"]
Jul 08 16:35:27.907054 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.906997    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:27.910392 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.910369    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 16:35:27.910670 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.910634    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 16:35:27.912857 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.912835    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 16:35:27.913088 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.913069    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 08 16:35:27.937331 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.937308    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f9489547d-l49zd"]
Jul 08 16:35:27.944007 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.943960    2575 patch_prober.go:28] interesting pod/image-registry-f9489547d-l49zd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 08 16:35:27.944211 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.944178    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f9489547d-l49zd" podUID="c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:35:27.968219 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.968037    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-4zfwc"]
Jul 08 16:35:27.983918 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.983895    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-4zfwc"
Jul 08 16:35:27.984120 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.984080    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-4zfwc"]
Jul 08 16:35:27.984204 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.984143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/024afa1f-ab08-4170-9806-47c769ec0560-klusterlet-config\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:27.984204 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.984194    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/024afa1f-ab08-4170-9806-47c769ec0560-tmp\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:27.984362 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.984233    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g62rs\" (UniqueName: \"kubernetes.io/projected/024afa1f-ab08-4170-9806-47c769ec0560-kube-api-access-g62rs\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:27.986617 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.986588    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 16:35:27.987418 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.987083    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 16:35:27.987418 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.987205    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7nxxk\""
Jul 08 16:35:27.999134 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:27.999110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:28.018055 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.018028    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4x4wh" event={"ID":"8751bafd-f26c-47bd-8fa8-f24755c1fb18","Type":"ContainerStarted","Data":"a0ec4192ca1454af8a9357828345d5011143bb02997191eceac5fd634df537a8"}
Jul 08 16:35:28.018163 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.018083    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4x4wh" event={"ID":"8751bafd-f26c-47bd-8fa8-f24755c1fb18","Type":"ContainerStarted","Data":"6ec009bb7a7c9567d23f8874e595fa977b4836fa5fc6ab2a4534b3fb9aec3c2a"}
Jul 08 16:35:28.042388 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.042341    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4x4wh" podStartSLOduration=67.60060991 podStartE2EDuration="1m9.042324749s" totalImagesPullingTime="1.441714839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:28.04098424 +0000 UTC m=+69.030541007" watchObservedRunningTime="2026-07-08 16:35:28.042324749 +0000 UTC m=+69.031881493"
Jul 08 16:35:28.068197 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.068175    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-759b5bcd4-7fjbt"]
Jul 08 16:35:28.085188 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.085162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/024afa1f-ab08-4170-9806-47c769ec0560-klusterlet-config\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:28.085294 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.085216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6qm9\" (UniqueName: \"kubernetes.io/projected/f66ead7a-f82f-4786-a47c-51e175651730-kube-api-access-t6qm9\") pod \"downloads-685bfb5f64-4zfwc\" (UID: \"f66ead7a-f82f-4786-a47c-51e175651730\") " pod="openshift-console/downloads-685bfb5f64-4zfwc"
Jul 08 16:35:28.085294 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.085249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/024afa1f-ab08-4170-9806-47c769ec0560-tmp\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:28.085294 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.085274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g62rs\" (UniqueName: \"kubernetes.io/projected/024afa1f-ab08-4170-9806-47c769ec0560-kube-api-access-g62rs\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:28.085777 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.085743    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/024afa1f-ab08-4170-9806-47c769ec0560-tmp\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:28.087532 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.087506    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-759b5bcd4-7fjbt"]
Jul 08 16:35:28.087628 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.087596    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.088126 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.088100    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/024afa1f-ab08-4170-9806-47c769ec0560-klusterlet-config\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:28.094501 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.094463    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:28.096235 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.096214    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g62rs\" (UniqueName: \"kubernetes.io/projected/024afa1f-ab08-4170-9806-47c769ec0560-kube-api-access-g62rs\") pod \"klusterlet-addon-workmgr-849f7b6dcf-j7mvl\" (UID: \"024afa1f-ab08-4170-9806-47c769ec0560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:28.180770 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.180703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:28.186563 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.186530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-image-registry-private-configuration\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.186721 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.186586    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-registry-certificates\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.186721 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.186605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lclsm\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-kube-api-access-lclsm\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.186721 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.186630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-installation-pull-secrets\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.186721 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.186674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6qm9\" (UniqueName: \"kubernetes.io/projected/f66ead7a-f82f-4786-a47c-51e175651730-kube-api-access-t6qm9\") pod \"downloads-685bfb5f64-4zfwc\" (UID: \"f66ead7a-f82f-4786-a47c-51e175651730\") " pod="openshift-console/downloads-685bfb5f64-4zfwc"
Jul 08 16:35:28.186721 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.186716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-bound-sa-token\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.186986 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.186959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-ca-trust-extracted\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.187044 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.187014    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-trusted-ca\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.187099 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.187051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-registry-tls\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.201157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.201138    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6qm9\" (UniqueName: \"kubernetes.io/projected/f66ead7a-f82f-4786-a47c-51e175651730-kube-api-access-t6qm9\") pod \"downloads-685bfb5f64-4zfwc\" (UID: \"f66ead7a-f82f-4786-a47c-51e175651730\") " pod="openshift-console/downloads-685bfb5f64-4zfwc"
Jul 08 16:35:28.222976 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.222952    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:28.273413 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.273353    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:28.288044 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-installation-pull-secrets\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.288207 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-bound-sa-token\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.288207 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-ca-trust-extracted\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.288207 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-trusted-ca\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.288207 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-registry-tls\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.288710 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-image-registry-private-configuration\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.288710 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-registry-certificates\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.288710 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288580    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-ca-trust-extracted\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.289102 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.288997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclsm\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-kube-api-access-lclsm\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.289328 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.289277    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-trusted-ca\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.289445 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.289395    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-registry-certificates\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.291580 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.291543    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-installation-pull-secrets\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.291580 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.291566    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-registry-tls\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.292001 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.291971    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-image-registry-private-configuration\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.295549 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.295530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-4zfwc"
Jul 08 16:35:28.297254 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.297231    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-bound-sa-token\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.297710 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.297685    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclsm\" (UniqueName: \"kubernetes.io/projected/5801ea7f-3647-45a1-8a15-72a7a5b9b8c6-kube-api-access-lclsm\") pod \"image-registry-759b5bcd4-7fjbt\" (UID: \"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6\") " pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.363409 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.363339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:28.411017 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.410962    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:28.592292 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.592206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:35:28.595002 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.594977    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afcc2e47-6a0a-498e-9e3d-3ab061a32e71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gd5xv\" (UID: \"afcc2e47-6a0a-498e-9e3d-3ab061a32e71\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:35:28.619105 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.619080    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jrh7q\""
Jul 08 16:35:28.627050 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.627029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"
Jul 08 16:35:28.793729 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.793694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:28.794383 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:28.794360    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edb90cb-255d-4637-9bd9-14e8c87c3c06-service-ca-bundle\") pod \"router-default-7f8778bf4f-tlgl8\" (UID: \"9edb90cb-255d-4637-9bd9-14e8c87c3c06\") " pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:29.049614 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:29.049584    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l74bm\""
Jul 08 16:35:29.057748 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:29.057729    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:30.309779 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:30.309737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:30.434479 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:30.434438    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.227980 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:31.227949    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f8778bf4f-tlgl8"]
Jul 08 16:35:31.231365 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:31.231310    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edb90cb_255d_4637_9bd9_14e8c87c3c06.slice/crio-4aeae3afbd0b58acaae2841c838b8ccea276f326b21e754ba55c971530874dfb WatchSource:0}: Error finding container 4aeae3afbd0b58acaae2841c838b8ccea276f326b21e754ba55c971530874dfb: Status 404 returned error can't find the container with id 4aeae3afbd0b58acaae2841c838b8ccea276f326b21e754ba55c971530874dfb
Jul 08 16:35:31.425757 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:31.425608    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"]
Jul 08 16:35:31.428550 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:31.428520    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv"]
Jul 08 16:35:31.428749 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:31.428706    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024afa1f_ab08_4170_9806_47c769ec0560.slice/crio-a2f3c394e40d15d7655f4d30f719d341da5728e193161456afa00f68edcd6962 WatchSource:0}: Error finding container a2f3c394e40d15d7655f4d30f719d341da5728e193161456afa00f68edcd6962: Status 404 returned error can't find the container with id a2f3c394e40d15d7655f4d30f719d341da5728e193161456afa00f68edcd6962
Jul 08 16:35:31.430568 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:31.430547    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcc2e47_6a0a_498e_9e3d_3ab061a32e71.slice/crio-873903e545d46aff5359c86051cc20285224ab3bcd92ef8a2842b5d68baa4b0b WatchSource:0}: Error finding container 873903e545d46aff5359c86051cc20285224ab3bcd92ef8a2842b5d68baa4b0b: Status 404 returned error can't find the container with id 873903e545d46aff5359c86051cc20285224ab3bcd92ef8a2842b5d68baa4b0b
Jul 08 16:35:31.473267 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:31.473235    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-4zfwc"]
Jul 08 16:35:31.474774 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:31.474750    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-759b5bcd4-7fjbt"]
Jul 08 16:35:31.477339 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:31.477305    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66ead7a_f82f_4786_a47c_51e175651730.slice/crio-dd5adde3477916c194966d895b19fda4d7df4e6271f6bae63fd26817549a39ed WatchSource:0}: Error finding container dd5adde3477916c194966d895b19fda4d7df4e6271f6bae63fd26817549a39ed: Status 404 returned error can't find the container with id dd5adde3477916c194966d895b19fda4d7df4e6271f6bae63fd26817549a39ed
Jul 08 16:35:31.478030 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:31.477934    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5801ea7f_3647_45a1_8a15_72a7a5b9b8c6.slice/crio-45e28a6bea605c572f237aea057c3666488522fd45fa060f9ff6cb92e83b3fc2 WatchSource:0}: Error finding container 45e28a6bea605c572f237aea057c3666488522fd45fa060f9ff6cb92e83b3fc2: Status 404 returned error can't find the container with id 45e28a6bea605c572f237aea057c3666488522fd45fa060f9ff6cb92e83b3fc2
Jul 08 16:35:32.039311 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.039271    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dtcdw" event={"ID":"2f1fa945-7def-4e82-9da9-78010a748562","Type":"ContainerStarted","Data":"98fd84cf151e3cddc69960776e5d51589a0d8efd8936af628137996963d759d2"}
Jul 08 16:35:32.039698 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.039607    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:35:32.041394 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.041317    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8" event={"ID":"9edb90cb-255d-4637-9bd9-14e8c87c3c06","Type":"ContainerStarted","Data":"21b78a2a82dfc00d645c51167e20057b5b6eb13f8fd4f8928dd59f644bbc9d4f"}
Jul 08 16:35:32.041394 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.041364    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8" event={"ID":"9edb90cb-255d-4637-9bd9-14e8c87c3c06","Type":"ContainerStarted","Data":"4aeae3afbd0b58acaae2841c838b8ccea276f326b21e754ba55c971530874dfb"}
Jul 08 16:35:32.042921 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.042855    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv" event={"ID":"afcc2e47-6a0a-498e-9e3d-3ab061a32e71","Type":"ContainerStarted","Data":"873903e545d46aff5359c86051cc20285224ab3bcd92ef8a2842b5d68baa4b0b"}
Jul 08 16:35:32.044288 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.044229    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-4zfwc" event={"ID":"f66ead7a-f82f-4786-a47c-51e175651730","Type":"ContainerStarted","Data":"dd5adde3477916c194966d895b19fda4d7df4e6271f6bae63fd26817549a39ed"}
Jul 08 16:35:32.046089 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.046063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v8x8z" event={"ID":"2a323b89-9740-4ab1-9d0d-1ca2bec3d93d","Type":"ContainerStarted","Data":"c621886bc5f0317ba8031cf3f3ce27149a52ca6b73656918f7bfbec4f5f298ad"}
Jul 08 16:35:32.047424 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.047398    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl" event={"ID":"024afa1f-ab08-4170-9806-47c769ec0560","Type":"ContainerStarted","Data":"a2f3c394e40d15d7655f4d30f719d341da5728e193161456afa00f68edcd6962"}
Jul 08 16:35:32.049744 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.049719    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt" event={"ID":"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6","Type":"ContainerStarted","Data":"21105184669c151a02721c0cc0f4b269d9db9c550575a519a8bae373d70049a1"}
Jul 08 16:35:32.049831 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.049764    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt" event={"ID":"5801ea7f-3647-45a1-8a15-72a7a5b9b8c6","Type":"ContainerStarted","Data":"45e28a6bea605c572f237aea057c3666488522fd45fa060f9ff6cb92e83b3fc2"}
Jul 08 16:35:32.049900 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.049880    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:32.058213 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.058191    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:32.061541 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.061522    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:32.064315 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.064275    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dtcdw" podStartSLOduration=67.886850173 podStartE2EDuration="1m13.064256887s" totalImagesPullingTime="5.177406714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:32.063538498 +0000 UTC m=+73.053095277" watchObservedRunningTime="2026-07-08 16:35:32.064256887 +0000 UTC m=+73.053813631"
Jul 08 16:35:32.108560 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.107987    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8" podStartSLOduration=36.107968224 podStartE2EDuration="36.107968224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:32.105717743 +0000 UTC m=+73.095274506" watchObservedRunningTime="2026-07-08 16:35:32.107968224 +0000 UTC m=+73.097524968"
Jul 08 16:35:32.108560 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.108132    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v8x8z" podStartSLOduration=67.681385166 podStartE2EDuration="1m13.108119815s" totalImagesPullingTime="5.426734649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:32.083159704 +0000 UTC m=+73.072716448" watchObservedRunningTime="2026-07-08 16:35:32.108119815 +0000 UTC m=+73.097676561"
Jul 08 16:35:32.131150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:32.131096    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt" podStartSLOduration=4.131083018 podStartE2EDuration="4.131083018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:32.1299786 +0000 UTC m=+73.119535370" watchObservedRunningTime="2026-07-08 16:35:32.131083018 +0000 UTC m=+73.120639740"
Jul 08 16:35:33.062227 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:33.057467    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:33.064318 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:33.064065    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f8778bf4f-tlgl8"
Jul 08 16:35:33.611607 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:33.611573    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.676367 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:33.676311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:34.091566 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:34.091460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:34.168320 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:34.168286    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:34.243101 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:34.243021    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:35.160605 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:35.160545    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:35.355101 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:35.355020    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:35.800632 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:35.800543    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:35.900323 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:35.900151    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:35.994360 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:35.994314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:36.072013 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:36.071928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv" event={"ID":"afcc2e47-6a0a-498e-9e3d-3ab061a32e71","Type":"ContainerStarted","Data":"cec8ccbd85b35fa60252805c8b38ab7798016b1cae6e3b800ed351adaf4da42f"}
Jul 08 16:35:36.074120 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:36.074084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl" event={"ID":"024afa1f-ab08-4170-9806-47c769ec0560","Type":"ContainerStarted","Data":"2c69814ebb8784e9711c990ecba787d86e4fd7c5fcf768174e1b19a9b7d7143a"}
Jul 08 16:35:36.074340 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:36.074321    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:36.076383 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:36.076363    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl"
Jul 08 16:35:36.093611 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:36.093565    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gd5xv" podStartSLOduration=36.067813814 podStartE2EDuration="40.093548155s" totalImagesPullingTime="4.025734341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:36.09220394 +0000 UTC m=+77.081760707" watchObservedRunningTime="2026-07-08 16:35:36.093548155 +0000 UTC m=+77.083104899"
Jul 08 16:35:36.109865 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:36.109815    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-849f7b6dcf-j7mvl" podStartSLOduration=5.066637468 podStartE2EDuration="9.109801228s" totalImagesPullingTime="4.04316376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:36.109177558 +0000 UTC m=+77.098734303" watchObservedRunningTime="2026-07-08 16:35:36.109801228 +0000 UTC m=+77.099357971"
Jul 08 16:35:36.212770 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:36.212709    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:37.053047 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:37.053010    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:37.171294 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:37.171246    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:37.943133 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:37.943107    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:39.200273 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:39.200234    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:39.340557 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:39.340524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:39.907686 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:39.907635    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:39.977331 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:39.977303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:41.024314 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:41.024166    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:41.094150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:41.094107    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:41.163340 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:41.163308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.213804 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:43.213771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.314493 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:43.314458    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.426223 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:43.426182    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.614811 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:43.614774    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:45.932424 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:45.932392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.031121 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.031088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.304132 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.303970    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.397867 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.397835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.539504 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.539472    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sxgqt"]
Jul 08 16:35:46.543430 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.543404    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.546335 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.546311    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 16:35:46.546899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.546877    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 16:35:46.547008 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.546937    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 16:35:46.547008 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.546958    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8djnp\""
Jul 08 16:35:46.548128 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.548107    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 16:35:46.589929 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.589869    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.638429 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-root\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.638527 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638441    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-textfile\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.638527 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-wtmp\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.638619 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.638619 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638558    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfzgb\" (UniqueName: \"kubernetes.io/projected/81c52002-1b8f-4e5c-be4d-88f75d65dc00-kube-api-access-vfzgb\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.638619 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-sys\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.638803 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638625    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81c52002-1b8f-4e5c-be4d-88f75d65dc00-metrics-client-ca\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.638803 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.638803 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.638777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-tls\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.694567 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.694536    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.739719 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-root\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.739833 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-textfile\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.739833 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-wtmp\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.739833 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739810    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-root\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.739979 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.739979 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739874    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-wtmp\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.739979 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzgb\" (UniqueName: \"kubernetes.io/projected/81c52002-1b8f-4e5c-be4d-88f75d65dc00-kube-api-access-vfzgb\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.739979 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-sys\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.739979 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739947    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81c52002-1b8f-4e5c-be4d-88f75d65dc00-metrics-client-ca\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.740221 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.739989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.740221 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.740030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-tls\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.740221 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.740032    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81c52002-1b8f-4e5c-be4d-88f75d65dc00-sys\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.740221 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.740146    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-textfile\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.740221 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:46.740155    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 08 16:35:46.740527 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:35:46.740236    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-tls podName:81c52002-1b8f-4e5c-be4d-88f75d65dc00 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:47.240214833 +0000 UTC m=+88.229771558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-tls") pod "node-exporter-sxgqt" (UID: "81c52002-1b8f-4e5c-be4d-88f75d65dc00") : secret "node-exporter-tls" not found
Jul 08 16:35:46.740677 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.740638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.740677 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.740638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81c52002-1b8f-4e5c-be4d-88f75d65dc00-metrics-client-ca\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.742456 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.742432    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.750816 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.750795    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzgb\" (UniqueName: \"kubernetes.io/projected/81c52002-1b8f-4e5c-be4d-88f75d65dc00-kube-api-access-vfzgb\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:46.792827 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.792799    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.891848 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.891817    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.958501 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:46.958472    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:47.244355 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:47.244275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-tls\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:47.247117 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:47.247087    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81c52002-1b8f-4e5c-be4d-88f75d65dc00-node-exporter-tls\") pod \"node-exporter-sxgqt\" (UID: \"81c52002-1b8f-4e5c-be4d-88f75d65dc00\") " pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:47.444337 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:47.444292    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:47.454118 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:47.454086    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sxgqt"
Jul 08 16:35:47.544784 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:47.544620    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:47.721165 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:47.721134    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.142122 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:48.142082    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.249475 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:48.249443    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:49.519717 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:49.519674    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:49.627801 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:49.627769    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:49.766251 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:49.766211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:50.339770 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:50.339732    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:50.450277 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:50.450230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:51.232441 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:51.232381    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:51.328268 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:51.328233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:51.428388 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:51.428350    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:52.487272 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:52.487231    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c52002_1b8f_4e5c_be4d_88f75d65dc00.slice/crio-dceabc8df95ebef5bf460826d9045c00e83cdc870840dd95de98529c5760dd7b WatchSource:0}: Error finding container dceabc8df95ebef5bf460826d9045c00e83cdc870840dd95de98529c5760dd7b: Status 404 returned error can't find the container with id dceabc8df95ebef5bf460826d9045c00e83cdc870840dd95de98529c5760dd7b
Jul 08 16:35:52.926453 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:52.926419    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:52.963913 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:52.963882    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f9489547d-l49zd" podUID="c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" containerName="registry" containerID="cri-o://d759eedffedbe3a49659c5f2a5a73d3db185c4be0bf965c4b3049a58b634ae24" gracePeriod=30
Jul 08 16:35:53.054064 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.054024    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:53.061188 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.061167    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-759b5bcd4-7fjbt"
Jul 08 16:35:53.144787 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.144702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-4zfwc" event={"ID":"f66ead7a-f82f-4786-a47c-51e175651730","Type":"ContainerStarted","Data":"524057ac3d2a99881b5aae35d231ed4afaddff5686137856cb12412f4745684d"}
Jul 08 16:35:53.145177 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.145129    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-4zfwc"
Jul 08 16:35:53.146357 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.146331    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxgqt" event={"ID":"81c52002-1b8f-4e5c-be4d-88f75d65dc00","Type":"ContainerStarted","Data":"dceabc8df95ebef5bf460826d9045c00e83cdc870840dd95de98529c5760dd7b"}
Jul 08 16:35:53.149039 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.149012    2575 generic.go:356] "Generic (PLEG): container finished" podID="c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" containerID="d759eedffedbe3a49659c5f2a5a73d3db185c4be0bf965c4b3049a58b634ae24" exitCode=0
Jul 08 16:35:53.149148 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.149072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f9489547d-l49zd" event={"ID":"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40","Type":"ContainerDied","Data":"d759eedffedbe3a49659c5f2a5a73d3db185c4be0bf965c4b3049a58b634ae24"}
Jul 08 16:35:53.152260 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.152221    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-4zfwc"
Jul 08 16:35:53.164426 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.164362    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-4zfwc" podStartSLOduration=4.78781081 podStartE2EDuration="26.164343992s" totalImagesPullingTime="21.376533182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:53.162285247 +0000 UTC m=+94.151841987" watchObservedRunningTime="2026-07-08 16:35:53.164343992 +0000 UTC m=+94.153900738"
Jul 08 16:35:53.228583 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.228216    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:53.244668 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.244626    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:53.294011 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.293981    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") pod \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") "
Jul 08 16:35:53.294191 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.294031    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-image-registry-private-configuration\") pod \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") "
Jul 08 16:35:53.294191 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.294105    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-installation-pull-secrets\") pod \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") "
Jul 08 16:35:53.294191 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.294130    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjxfz\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-kube-api-access-tjxfz\") pod \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") "
Jul 08 16:35:53.294191 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.294168    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-trusted-ca\") pod \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") "
Jul 08 16:35:53.294428 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.294224    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-bound-sa-token\") pod \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") "
Jul 08 16:35:53.294428 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.294248    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-certificates\") pod \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") "
Jul 08 16:35:53.294428 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.294293    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-ca-trust-extracted\") pod \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\" (UID: \"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40\") "
Jul 08 16:35:53.296633 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.296340    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:53.296633 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.296567    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:53.300158 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.300128    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:53.300452 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.300422    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:53.300731 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.300707    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:35:53.301768 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.301709    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:35:53.301768 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.301783    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-kube-api-access-tjxfz" (OuterVolumeSpecName: "kube-api-access-tjxfz") pod "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40"). InnerVolumeSpecName "kube-api-access-tjxfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:35:53.306169 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.306129    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" (UID: "c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:35:53.389744 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.389703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:53.395975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.395884    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-trusted-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:35:53.395975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.395914    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-bound-sa-token\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:35:53.395975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.395929    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-certificates\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:35:53.395975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.395944    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-ca-trust-extracted\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:35:53.395975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.395958    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-registry-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:35:53.395975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.395973    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-image-registry-private-configuration\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:35:53.395975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.395987    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-installation-pull-secrets\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:35:53.396493 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.396001    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxfz\" (UniqueName: \"kubernetes.io/projected/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40-kube-api-access-tjxfz\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:35:53.494916 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:53.494882    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:54.154870 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:54.154842    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9489547d-l49zd"
Jul 08 16:35:54.155070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:54.154839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f9489547d-l49zd" event={"ID":"c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40","Type":"ContainerDied","Data":"0481526e148ffbd8058235c22b5f75dff8b6d9308d6dccc8a43ffec98c3b9218"}
Jul 08 16:35:54.155070 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:54.154993    2575 scope.go:117] "RemoveContainer" containerID="d759eedffedbe3a49659c5f2a5a73d3db185c4be0bf965c4b3049a58b634ae24"
Jul 08 16:35:54.180324 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:54.180297    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f9489547d-l49zd"]
Jul 08 16:35:54.189472 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:54.189446    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f9489547d-l49zd"]
Jul 08 16:35:54.215555 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:54.215529    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:54.299016 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:54.298984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:55.162215 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.162181    2575 generic.go:356] "Generic (PLEG): container finished" podID="81c52002-1b8f-4e5c-be4d-88f75d65dc00" containerID="bf35701ccc6b427b9ad3768fda8afbd2dd08724c0b967952ca5ebaa8a24f0266" exitCode=0
Jul 08 16:35:55.162807 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.162269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxgqt" event={"ID":"81c52002-1b8f-4e5c-be4d-88f75d65dc00","Type":"ContainerDied","Data":"bf35701ccc6b427b9ad3768fda8afbd2dd08724c0b967952ca5ebaa8a24f0266"}
Jul 08 16:35:55.546838 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.546796    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:55.608706 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.608676    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" path="/var/lib/kubelet/pods/c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40/volumes"
Jul 08 16:35:55.645231 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.645202    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:55.969868 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.969835    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65fcc755bf-6jbld"]
Jul 08 16:35:55.970218 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.970201    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" containerName="registry"
Jul 08 16:35:55.970218 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.970218    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" containerName="registry"
Jul 08 16:35:55.970334 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:55.970284    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a38bfc-b69f-48f0-a6c0-e6a793ad9c40" containerName="registry"
Jul 08 16:35:56.001524 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.001500    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fcc755bf-6jbld"]
Jul 08 16:35:56.001661 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.001608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.004337 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.004306    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 16:35:56.005882 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.005441    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2bkm5\""
Jul 08 16:35:56.005882 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.005715    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 16:35:56.005882 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.005775    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 16:35:56.005882 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.005803    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 16:35:56.006153 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.005998    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 16:35:56.010756 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.010736    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 16:35:56.122515 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.122484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-trusted-ca-bundle\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.122679 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.122535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-oauth-serving-cert\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.122679 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.122572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-serving-cert\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.122679 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.122624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-oauth-config\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.122679 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.122671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-console-config\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.122884 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.122704    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-service-ca\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.122884 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.122763    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvcx\" (UniqueName: \"kubernetes.io/projected/002a2b44-5314-4104-aa2e-f51ea9438126-kube-api-access-8fvcx\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.169215 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.169183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxgqt" event={"ID":"81c52002-1b8f-4e5c-be4d-88f75d65dc00","Type":"ContainerStarted","Data":"749e7a84bebb446eb2e4eb450f82d4914a8629f207158d1bc5e472d9ddedca52"}
Jul 08 16:35:56.169605 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.169249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxgqt" event={"ID":"81c52002-1b8f-4e5c-be4d-88f75d65dc00","Type":"ContainerStarted","Data":"627c508fd784e4723a154959e521854d5698176f57ec956edcc38ec00ea1a943"}
Jul 08 16:35:56.188175 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.188120    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sxgqt" podStartSLOduration=8.373794313 podStartE2EDuration="10.188102274s" totalImagesPullingTime="1.725065833s" totalInitContainerRuntime="89.242128ms" isStatefulPod=true podCreationTimestamp="2026-07-08 16:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:56.187795868 +0000 UTC m=+97.177352612" watchObservedRunningTime="2026-07-08 16:35:56.188102274 +0000 UTC m=+97.177659018"
Jul 08 16:35:56.223350 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.223260    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-oauth-serving-cert\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.223350 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.223322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-serving-cert\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.223551 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.223382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-oauth-config\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.223551 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.223409    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-console-config\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.223551 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.223451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-service-ca\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.223551 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.223516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvcx\" (UniqueName: \"kubernetes.io/projected/002a2b44-5314-4104-aa2e-f51ea9438126-kube-api-access-8fvcx\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.223551 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.223546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-trusted-ca-bundle\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.224451 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.224056    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-oauth-serving-cert\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.224451 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.224245    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-service-ca\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.224631 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.224547    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-console-config\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.224722 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.224626    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-trusted-ca-bundle\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.226266 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.226241    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-serving-cert\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.226863 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.226842    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-oauth-config\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.232272 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.232248    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvcx\" (UniqueName: \"kubernetes.io/projected/002a2b44-5314-4104-aa2e-f51ea9438126-kube-api-access-8fvcx\") pod \"console-65fcc755bf-6jbld\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") " pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.313399 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.313372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:35:56.510822 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:56.510796    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fcc755bf-6jbld"]
Jul 08 16:35:56.513131 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:35:56.513100    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002a2b44_5314_4104_aa2e_f51ea9438126.slice/crio-169f90230ca95ee769fdc8cbda58a067e1f624db9df8d7a903ac2c75faff3855 WatchSource:0}: Error finding container 169f90230ca95ee769fdc8cbda58a067e1f624db9df8d7a903ac2c75faff3855: Status 404 returned error can't find the container with id 169f90230ca95ee769fdc8cbda58a067e1f624db9df8d7a903ac2c75faff3855
Jul 08 16:35:57.175140 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:35:57.175101    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fcc755bf-6jbld" event={"ID":"002a2b44-5314-4104-aa2e-f51ea9438126","Type":"ContainerStarted","Data":"169f90230ca95ee769fdc8cbda58a067e1f624db9df8d7a903ac2c75faff3855"}
Jul 08 16:36:00.109229 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:00.109169    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:00.173396 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:00.173329    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:01.196121 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:01.196080    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fcc755bf-6jbld" event={"ID":"002a2b44-5314-4104-aa2e-f51ea9438126","Type":"ContainerStarted","Data":"38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e"}
Jul 08 16:36:01.215391 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:01.215343    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65fcc755bf-6jbld" podStartSLOduration=2.463212618 podStartE2EDuration="6.215323036s" totalImagesPullingTime="3.752110418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:01.213610319 +0000 UTC m=+102.203167101" watchObservedRunningTime="2026-07-08 16:36:01.215323036 +0000 UTC m=+102.204879782"
Jul 08 16:36:03.059471 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:03.059440    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dtcdw"
Jul 08 16:36:03.167134 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:03.167056    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:03.231792 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:03.231758    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:06.314332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:06.314296    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:36:06.314733 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:06.314357    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:36:06.318815 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:06.318798    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:36:07.219313 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:07.219288    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:36:07.414031 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:07.414002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.514607 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:07.514502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.615862 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:07.615839    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.921777 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:07.921746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:08.003397 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:08.003357    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:14.742992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:14.742950    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:14.838171 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:14.838138    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:17.847570 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:17.847533    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:17.945720 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:17.945692    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:18.233144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:18.233116    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:21.268016 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:21.267988    2575 generic.go:356] "Generic (PLEG): container finished" podID="8319143d-9ddf-4074-b9ef-1acaffa068f0" containerID="d4c141fbed7d1ee000aa1b309cd4b055ebb82ff8281b1da5d9d8ef0025fa4a9b" exitCode=0
Jul 08 16:36:21.268331 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:21.268039    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45" event={"ID":"8319143d-9ddf-4074-b9ef-1acaffa068f0","Type":"ContainerDied","Data":"d4c141fbed7d1ee000aa1b309cd4b055ebb82ff8281b1da5d9d8ef0025fa4a9b"}
Jul 08 16:36:21.268374 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:21.268354    2575 scope.go:117] "RemoveContainer" containerID="d4c141fbed7d1ee000aa1b309cd4b055ebb82ff8281b1da5d9d8ef0025fa4a9b"
Jul 08 16:36:22.273258 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:22.273226    2575 generic.go:356] "Generic (PLEG): container finished" podID="650f88a0-c7d3-44d7-b457-d81d1584808c" containerID="f3eecc8cd24338c1772a6cbb1049505e5e13c35c5dc5f74b73b31ba4e1c337e5" exitCode=0
Jul 08 16:36:22.273749 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:22.273308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8" event={"ID":"650f88a0-c7d3-44d7-b457-d81d1584808c","Type":"ContainerDied","Data":"f3eecc8cd24338c1772a6cbb1049505e5e13c35c5dc5f74b73b31ba4e1c337e5"}
Jul 08 16:36:22.273749 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:22.273744    2575 scope.go:117] "RemoveContainer" containerID="f3eecc8cd24338c1772a6cbb1049505e5e13c35c5dc5f74b73b31ba4e1c337e5"
Jul 08 16:36:22.275163 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:22.275140    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tmb45" event={"ID":"8319143d-9ddf-4074-b9ef-1acaffa068f0","Type":"ContainerStarted","Data":"65a2040973d0a99cfffd4ffc67173eda058a9aeca6e518c72f9344abb94ccd9d"}
Jul 08 16:36:22.975402 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:22.975364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:23.098108 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:23.098076    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:23.280773 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:36:23.280701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-mdgr8" event={"ID":"650f88a0-c7d3-44d7-b457-d81d1584808c","Type":"ContainerStarted","Data":"161db5a5159bb578692d02715e30aa1f27c219b868ea30cfe2d598ab0ff398e6"}
Jul 08 16:37:05.860149 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:05.860106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:05.977386 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:05.977350    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:15.372822 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.372742    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6477866558-l2n74"]
Jul 08 16:37:15.375304 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.375282    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.386589 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.386558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6477866558-l2n74"]
Jul 08 16:37:15.397283 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.397258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-serving-cert\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.397392 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.397294    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-oauth-config\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.397392 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.397318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-oauth-serving-cert\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.397392 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.397374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-console-config\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.397513 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.397418    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-service-ca\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.397513 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.397447    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/ce026e29-4474-424d-aa26-af2074fee271-kube-api-access-ncmgr\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.397513 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.397505    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-trusted-ca-bundle\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.452052 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.452023    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:15.498215 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.498178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-serving-cert\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.498330 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.498226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-oauth-config\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.498330 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.498261    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-oauth-serving-cert\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.498330 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.498285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-console-config\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.498330 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.498311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-service-ca\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.498538 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.498337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/ce026e29-4474-424d-aa26-af2074fee271-kube-api-access-ncmgr\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.498538 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.498377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-trusted-ca-bundle\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.499193 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.499168    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-console-config\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.499271 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.499191    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-service-ca\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.499271 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.499205    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-oauth-serving-cert\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.499344 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.499303    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-trusted-ca-bundle\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.500768 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.500745    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-oauth-config\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.500902 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.500886    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-serving-cert\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.505393 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.505378    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/ce026e29-4474-424d-aa26-af2074fee271-kube-api-access-ncmgr\") pod \"console-6477866558-l2n74\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") " pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.552312 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.552288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:15.651976 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.651914    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:15.684861 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.684840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:15.810914 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:15.810891    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6477866558-l2n74"]
Jul 08 16:37:15.813324 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:37:15.813300    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce026e29_4474_424d_aa26_af2074fee271.slice/crio-df8134aa4fcb1ffe76cb13b21bb7f2f36d95da7c4dd9680bb8c6cb6f6214ec80 WatchSource:0}: Error finding container df8134aa4fcb1ffe76cb13b21bb7f2f36d95da7c4dd9680bb8c6cb6f6214ec80: Status 404 returned error can't find the container with id df8134aa4fcb1ffe76cb13b21bb7f2f36d95da7c4dd9680bb8c6cb6f6214ec80
Jul 08 16:37:16.464424 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:16.464391    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477866558-l2n74" event={"ID":"ce026e29-4474-424d-aa26-af2074fee271","Type":"ContainerStarted","Data":"0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c"}
Jul 08 16:37:16.464921 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:16.464447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477866558-l2n74" event={"ID":"ce026e29-4474-424d-aa26-af2074fee271","Type":"ContainerStarted","Data":"df8134aa4fcb1ffe76cb13b21bb7f2f36d95da7c4dd9680bb8c6cb6f6214ec80"}
Jul 08 16:37:16.483807 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:16.483759    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6477866558-l2n74" podStartSLOduration=1.483743753 podStartE2EDuration="1.483743753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:37:16.481992085 +0000 UTC m=+177.471548845" watchObservedRunningTime="2026-07-08 16:37:16.483743753 +0000 UTC m=+177.473300497"
Jul 08 16:37:16.622111 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:16.622082    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:16.812318 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:16.812244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:25.685954 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:25.685918    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:25.686337 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:25.685969    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:25.690584 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:25.690563    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:26.502899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:26.502868    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:37:26.555716 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:26.555687    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fcc755bf-6jbld"]
Jul 08 16:37:26.694552 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:26.694516    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:26.795906 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:26.795822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:26.857237 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:26.857206    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:51.577737 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.577688    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-65fcc755bf-6jbld" podUID="002a2b44-5314-4104-aa2e-f51ea9438126" containerName="console" containerID="cri-o://38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e" gracePeriod=15
Jul 08 16:37:51.816827 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.816804    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:37:51.847880 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.847824    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-service-ca\") pod \"002a2b44-5314-4104-aa2e-f51ea9438126\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") "
Jul 08 16:37:51.847880 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.847863    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-oauth-serving-cert\") pod \"002a2b44-5314-4104-aa2e-f51ea9438126\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") "
Jul 08 16:37:51.848019 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.847886    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-oauth-config\") pod \"002a2b44-5314-4104-aa2e-f51ea9438126\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") "
Jul 08 16:37:51.848019 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.847920    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-trusted-ca-bundle\") pod \"002a2b44-5314-4104-aa2e-f51ea9438126\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") "
Jul 08 16:37:51.848019 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.847947    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-console-config\") pod \"002a2b44-5314-4104-aa2e-f51ea9438126\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") "
Jul 08 16:37:51.848019 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.847976    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-serving-cert\") pod \"002a2b44-5314-4104-aa2e-f51ea9438126\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") "
Jul 08 16:37:51.848019 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.848004    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fvcx\" (UniqueName: \"kubernetes.io/projected/002a2b44-5314-4104-aa2e-f51ea9438126-kube-api-access-8fvcx\") pod \"002a2b44-5314-4104-aa2e-f51ea9438126\" (UID: \"002a2b44-5314-4104-aa2e-f51ea9438126\") "
Jul 08 16:37:51.848390 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.848241    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-service-ca" (OuterVolumeSpecName: "service-ca") pod "002a2b44-5314-4104-aa2e-f51ea9438126" (UID: "002a2b44-5314-4104-aa2e-f51ea9438126"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:51.848390 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.848361    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "002a2b44-5314-4104-aa2e-f51ea9438126" (UID: "002a2b44-5314-4104-aa2e-f51ea9438126"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:51.848564 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.848403    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-console-config" (OuterVolumeSpecName: "console-config") pod "002a2b44-5314-4104-aa2e-f51ea9438126" (UID: "002a2b44-5314-4104-aa2e-f51ea9438126"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:51.848564 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.848553    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "002a2b44-5314-4104-aa2e-f51ea9438126" (UID: "002a2b44-5314-4104-aa2e-f51ea9438126"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:51.850175 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.850134    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "002a2b44-5314-4104-aa2e-f51ea9438126" (UID: "002a2b44-5314-4104-aa2e-f51ea9438126"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:51.850258 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.850177    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "002a2b44-5314-4104-aa2e-f51ea9438126" (UID: "002a2b44-5314-4104-aa2e-f51ea9438126"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:51.850258 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.850240    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002a2b44-5314-4104-aa2e-f51ea9438126-kube-api-access-8fvcx" (OuterVolumeSpecName: "kube-api-access-8fvcx") pod "002a2b44-5314-4104-aa2e-f51ea9438126" (UID: "002a2b44-5314-4104-aa2e-f51ea9438126"). InnerVolumeSpecName "kube-api-access-8fvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:37:51.949399 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.949376    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-trusted-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:37:51.949399 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.949397    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-console-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:37:51.949554 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.949408    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:37:51.949554 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.949418    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fvcx\" (UniqueName: \"kubernetes.io/projected/002a2b44-5314-4104-aa2e-f51ea9438126-kube-api-access-8fvcx\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:37:51.949554 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.949426    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:37:51.949554 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.949434    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002a2b44-5314-4104-aa2e-f51ea9438126-oauth-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:37:51.949554 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:51.949442    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002a2b44-5314-4104-aa2e-f51ea9438126-console-oauth-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:37:52.587533 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.587501    2575 generic.go:356] "Generic (PLEG): container finished" podID="002a2b44-5314-4104-aa2e-f51ea9438126" containerID="38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e" exitCode=0
Jul 08 16:37:52.587956 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.587541    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fcc755bf-6jbld"
Jul 08 16:37:52.587956 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.587586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fcc755bf-6jbld" event={"ID":"002a2b44-5314-4104-aa2e-f51ea9438126","Type":"ContainerDied","Data":"38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e"}
Jul 08 16:37:52.587956 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.587634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fcc755bf-6jbld" event={"ID":"002a2b44-5314-4104-aa2e-f51ea9438126","Type":"ContainerDied","Data":"169f90230ca95ee769fdc8cbda58a067e1f624db9df8d7a903ac2c75faff3855"}
Jul 08 16:37:52.587956 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.587674    2575 scope.go:117] "RemoveContainer" containerID="38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e"
Jul 08 16:37:52.596243 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.596227    2575 scope.go:117] "RemoveContainer" containerID="38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e"
Jul 08 16:37:52.596517 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:37:52.596496    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e\": container with ID starting with 38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e not found: ID does not exist" containerID="38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e"
Jul 08 16:37:52.596566 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.596525    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e"} err="failed to get container status \"38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e\": rpc error: code = NotFound desc = could not find container \"38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e\": container with ID starting with 38c0d04cd784dea2d3342c69d8fda318cac483dacf119586b1d63f1197402b2e not found: ID does not exist"
Jul 08 16:37:52.607207 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.607186    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fcc755bf-6jbld"]
Jul 08 16:37:52.610418 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:52.610400    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65fcc755bf-6jbld"]
Jul 08 16:37:53.096922 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:53.096895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:53.184090 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:53.184060    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:53.608299 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:37:53.608274    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="002a2b44-5314-4104-aa2e-f51ea9438126" path="/var/lib/kubelet/pods/002a2b44-5314-4104-aa2e-f51ea9438126/volumes"
Jul 08 16:38:21.160866 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:21.160835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:21.230005 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:21.229973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:24.333280 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.333249    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69b5d56455-c252j"]
Jul 08 16:38:24.333683 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.333507    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="002a2b44-5314-4104-aa2e-f51ea9438126" containerName="console"
Jul 08 16:38:24.333683 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.333517    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="002a2b44-5314-4104-aa2e-f51ea9438126" containerName="console"
Jul 08 16:38:24.333683 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.333559    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="002a2b44-5314-4104-aa2e-f51ea9438126" containerName="console"
Jul 08 16:38:24.335996 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.335972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.349985 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.349965    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69b5d56455-c252j"]
Jul 08 16:38:24.394385 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.394355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:24.477124 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.477104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-oauth-config\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.477237 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.477135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-serving-cert\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.477237 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.477157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-oauth-serving-cert\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.477237 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.477182    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-config\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.477237 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.477198    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-service-ca\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.477386 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.477257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-trusted-ca-bundle\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.477386 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.477281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qdxb\" (UniqueName: \"kubernetes.io/projected/cf041cca-8f0a-40be-88ee-634f3eaf40bf-kube-api-access-6qdxb\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.578296 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.578264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-oauth-serving-cert\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.578296 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.578303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-config\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.578516 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.578317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-service-ca\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.578516 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.578366    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-trusted-ca-bundle\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.578516 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.578399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdxb\" (UniqueName: \"kubernetes.io/projected/cf041cca-8f0a-40be-88ee-634f3eaf40bf-kube-api-access-6qdxb\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.578516 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.578431    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-oauth-config\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.578516 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.578462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-serving-cert\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.579376 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.579348    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-service-ca\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.579498 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.579480    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-oauth-serving-cert\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.579571 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.579492    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-config\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.579703 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.579683    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-trusted-ca-bundle\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.583103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.583083    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-serving-cert\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.585488 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.585437    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-oauth-config\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.586426 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.586405    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdxb\" (UniqueName: \"kubernetes.io/projected/cf041cca-8f0a-40be-88ee-634f3eaf40bf-kube-api-access-6qdxb\") pod \"console-69b5d56455-c252j\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") " pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.646834 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.646811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:24.770585 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:24.770558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69b5d56455-c252j"]
Jul 08 16:38:24.772334 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:38:24.772309    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf041cca_8f0a_40be_88ee_634f3eaf40bf.slice/crio-a584ed129edfee5755cf30566bf7bc69a5151aa045939c476c046cbd94d5ce13 WatchSource:0}: Error finding container a584ed129edfee5755cf30566bf7bc69a5151aa045939c476c046cbd94d5ce13: Status 404 returned error can't find the container with id a584ed129edfee5755cf30566bf7bc69a5151aa045939c476c046cbd94d5ce13
Jul 08 16:38:25.704725 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:25.704692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b5d56455-c252j" event={"ID":"cf041cca-8f0a-40be-88ee-634f3eaf40bf","Type":"ContainerStarted","Data":"d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137"}
Jul 08 16:38:25.705085 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:25.704736    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b5d56455-c252j" event={"ID":"cf041cca-8f0a-40be-88ee-634f3eaf40bf","Type":"ContainerStarted","Data":"a584ed129edfee5755cf30566bf7bc69a5151aa045939c476c046cbd94d5ce13"}
Jul 08 16:38:25.722571 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:25.722527    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69b5d56455-c252j" podStartSLOduration=1.7225131999999999 podStartE2EDuration="1.7225132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:25.721590042 +0000 UTC m=+246.711146823" watchObservedRunningTime="2026-07-08 16:38:25.7225132 +0000 UTC m=+246.712069947"
Jul 08 16:38:28.960560 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:28.960531    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:33.218947 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:33.218917    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:33.301982 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:33.301952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:34.647095 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:34.647069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:34.647469 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:34.647114    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:34.651831 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:34.651803    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:34.737199 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:34.737175    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:38:34.788418 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:34.788393    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6477866558-l2n74"]
Jul 08 16:38:34.896815 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:34.896779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:34.998091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:34.998007    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:35.058713 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:35.058685    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:36.949921 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:36.949884    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:37.936171 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:37.936134    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:38.036200 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:38.036178    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:59.809072 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:38:59.808979    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6477866558-l2n74" podUID="ce026e29-4474-424d-aa26-af2074fee271" containerName="console" containerID="cri-o://0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c" gracePeriod=15
Jul 08 16:39:00.038987 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.038965    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:39:00.225984 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.225944    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-serving-cert\") pod \"ce026e29-4474-424d-aa26-af2074fee271\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") "
Jul 08 16:39:00.226157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.225998    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-oauth-serving-cert\") pod \"ce026e29-4474-424d-aa26-af2074fee271\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") "
Jul 08 16:39:00.226157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226020    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-trusted-ca-bundle\") pod \"ce026e29-4474-424d-aa26-af2074fee271\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") "
Jul 08 16:39:00.226157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226064    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/ce026e29-4474-424d-aa26-af2074fee271-kube-api-access-ncmgr\") pod \"ce026e29-4474-424d-aa26-af2074fee271\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") "
Jul 08 16:39:00.226157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226082    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-console-config\") pod \"ce026e29-4474-424d-aa26-af2074fee271\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") "
Jul 08 16:39:00.226157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226098    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-oauth-config\") pod \"ce026e29-4474-424d-aa26-af2074fee271\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") "
Jul 08 16:39:00.226157 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226114    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-service-ca\") pod \"ce026e29-4474-424d-aa26-af2074fee271\" (UID: \"ce026e29-4474-424d-aa26-af2074fee271\") "
Jul 08 16:39:00.226568 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226536    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ce026e29-4474-424d-aa26-af2074fee271" (UID: "ce026e29-4474-424d-aa26-af2074fee271"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:39:00.226713 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226559    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-console-config" (OuterVolumeSpecName: "console-config") pod "ce026e29-4474-424d-aa26-af2074fee271" (UID: "ce026e29-4474-424d-aa26-af2074fee271"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:39:00.226713 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226566    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ce026e29-4474-424d-aa26-af2074fee271" (UID: "ce026e29-4474-424d-aa26-af2074fee271"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:39:00.226713 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.226594    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-service-ca" (OuterVolumeSpecName: "service-ca") pod "ce026e29-4474-424d-aa26-af2074fee271" (UID: "ce026e29-4474-424d-aa26-af2074fee271"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:39:00.228096 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.228070    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ce026e29-4474-424d-aa26-af2074fee271" (UID: "ce026e29-4474-424d-aa26-af2074fee271"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:39:00.228191 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.228175    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ce026e29-4474-424d-aa26-af2074fee271" (UID: "ce026e29-4474-424d-aa26-af2074fee271"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:39:00.228236 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.228190    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce026e29-4474-424d-aa26-af2074fee271-kube-api-access-ncmgr" (OuterVolumeSpecName: "kube-api-access-ncmgr") pod "ce026e29-4474-424d-aa26-af2074fee271" (UID: "ce026e29-4474-424d-aa26-af2074fee271"). InnerVolumeSpecName "kube-api-access-ncmgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:39:00.326657 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.326625    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-trusted-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:39:00.326740 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.326663    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/ce026e29-4474-424d-aa26-af2074fee271-kube-api-access-ncmgr\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:39:00.326740 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.326673    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-console-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:39:00.326740 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.326681    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-oauth-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:39:00.326740 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.326689    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:39:00.326740 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.326697    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce026e29-4474-424d-aa26-af2074fee271-console-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:39:00.326740 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.326705    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce026e29-4474-424d-aa26-af2074fee271-oauth-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:39:00.826243 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.826217    2575 generic.go:356] "Generic (PLEG): container finished" podID="ce026e29-4474-424d-aa26-af2074fee271" containerID="0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c" exitCode=0
Jul 08 16:39:00.826582 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.826261    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477866558-l2n74"
Jul 08 16:39:00.826582 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.826268    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477866558-l2n74" event={"ID":"ce026e29-4474-424d-aa26-af2074fee271","Type":"ContainerDied","Data":"0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c"}
Jul 08 16:39:00.826582 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.826305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477866558-l2n74" event={"ID":"ce026e29-4474-424d-aa26-af2074fee271","Type":"ContainerDied","Data":"df8134aa4fcb1ffe76cb13b21bb7f2f36d95da7c4dd9680bb8c6cb6f6214ec80"}
Jul 08 16:39:00.826582 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.826324    2575 scope.go:117] "RemoveContainer" containerID="0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c"
Jul 08 16:39:00.835352 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.835334    2575 scope.go:117] "RemoveContainer" containerID="0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c"
Jul 08 16:39:00.835593 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:39:00.835577    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c\": container with ID starting with 0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c not found: ID does not exist" containerID="0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c"
Jul 08 16:39:00.835630 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.835600    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c"} err="failed to get container status \"0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c\": rpc error: code = NotFound desc = could not find container \"0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c\": container with ID starting with 0560ffb116cb13ecad640b4e90a54762abfbdda2d26a2862a91a17599b62fd6c not found: ID does not exist"
Jul 08 16:39:00.847899 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.847875    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6477866558-l2n74"]
Jul 08 16:39:00.851901 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:00.851880    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6477866558-l2n74"]
Jul 08 16:39:01.607686 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:01.607635    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce026e29-4474-424d-aa26-af2074fee271" path="/var/lib/kubelet/pods/ce026e29-4474-424d-aa26-af2074fee271/volumes"
Jul 08 16:39:03.046332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:03.046296    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:19.424015 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:19.423985    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:39:19.427208 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:19.427186    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 16:39:19.431564 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:39:19.431547    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:40:24.488252 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:40:24.488216    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:33.838133 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:33.838097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:33.950342 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:33.950314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:37.227144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:37.227101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:37.329572 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:37.329540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:37.604602 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:37.604571    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:37.702258 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:37.702232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:37.815333 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:37.815304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:38.591726 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.591693    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"]
Jul 08 16:41:38.592079 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.591964    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce026e29-4474-424d-aa26-af2074fee271" containerName="console"
Jul 08 16:41:38.592079 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.591975    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce026e29-4474-424d-aa26-af2074fee271" containerName="console"
Jul 08 16:41:38.592079 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.592032    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce026e29-4474-424d-aa26-af2074fee271" containerName="console"
Jul 08 16:41:38.594027 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.594011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.596197 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.596172    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nf5xz\""
Jul 08 16:41:38.596395 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.596380    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 08 16:41:38.597270 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.597251    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 08 16:41:38.612790 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.612769    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"]
Jul 08 16:41:38.692582 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.692561    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.692706 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.692592    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.692706 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.692620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6gqf\" (UniqueName: \"kubernetes.io/projected/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-kube-api-access-g6gqf\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.793134 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.793097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.793231 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.793153    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6gqf\" (UniqueName: \"kubernetes.io/projected/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-kube-api-access-g6gqf\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.793231 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.793203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.793481 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.793462    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.793519 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.793478    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.801593 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.801566    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6gqf\" (UniqueName: \"kubernetes.io/projected/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-kube-api-access-g6gqf\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:38.903571 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:38.903547    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:41:39.029271 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:39.029249    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"]
Jul 08 16:41:39.031297 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:41:39.031269    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd367cb4e_fb71_4b83_9ce6_9d05fa67fd3a.slice/crio-009b65f1417b9680c3fa66ca432ff88be79dc0b8b3037a28d2b26c5e496be386 WatchSource:0}: Error finding container 009b65f1417b9680c3fa66ca432ff88be79dc0b8b3037a28d2b26c5e496be386: Status 404 returned error can't find the container with id 009b65f1417b9680c3fa66ca432ff88be79dc0b8b3037a28d2b26c5e496be386
Jul 08 16:41:39.033190 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:39.033172    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:41:39.377736 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:39.377708    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh" event={"ID":"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a","Type":"ContainerStarted","Data":"009b65f1417b9680c3fa66ca432ff88be79dc0b8b3037a28d2b26c5e496be386"}
Jul 08 16:41:46.408437 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:46.408409    2575 generic.go:356] "Generic (PLEG): container finished" podID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerID="e69913b37617851ffca587188d4b580e3f3dd736f4778dd49f627f1fc30e59a3" exitCode=0
Jul 08 16:41:46.408742 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:46.408471    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh" event={"ID":"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a","Type":"ContainerDied","Data":"e69913b37617851ffca587188d4b580e3f3dd736f4778dd49f627f1fc30e59a3"}
Jul 08 16:41:49.422376 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:49.422347    2575 generic.go:356] "Generic (PLEG): container finished" podID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerID="ca38045b27a1d7845304a58d8712b51761a7b551ae17c6a0b54507d9a9baa3d2" exitCode=0
Jul 08 16:41:49.422757 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:49.422440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh" event={"ID":"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a","Type":"ContainerDied","Data":"ca38045b27a1d7845304a58d8712b51761a7b551ae17c6a0b54507d9a9baa3d2"}
Jul 08 16:41:59.462530 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:59.462459    2575 generic.go:356] "Generic (PLEG): container finished" podID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerID="ccf5cd5c068832c6021c8f83bf1830c31e9f242b9b5f04f2d5e3c233a5f6132a" exitCode=0
Jul 08 16:41:59.462856 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:41:59.462533    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh" event={"ID":"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a","Type":"ContainerDied","Data":"ccf5cd5c068832c6021c8f83bf1830c31e9f242b9b5f04f2d5e3c233a5f6132a"}
Jul 08 16:42:00.631485 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.631459    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:42:00.758501 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.758439    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-util\") pod \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") "
Jul 08 16:42:00.758501 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.758477    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6gqf\" (UniqueName: \"kubernetes.io/projected/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-kube-api-access-g6gqf\") pod \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") "
Jul 08 16:42:00.758680 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.758534    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-bundle\") pod \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\" (UID: \"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a\") "
Jul 08 16:42:00.759199 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.759175    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-bundle" (OuterVolumeSpecName: "bundle") pod "d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" (UID: "d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:00.760636 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.760603    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-kube-api-access-g6gqf" (OuterVolumeSpecName: "kube-api-access-g6gqf") pod "d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" (UID: "d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a"). InnerVolumeSpecName "kube-api-access-g6gqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:42:00.762818 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.762798    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-util" (OuterVolumeSpecName: "util") pod "d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" (UID: "d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:00.859775 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.859755    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-util\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:00.859775 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.859776    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6gqf\" (UniqueName: \"kubernetes.io/projected/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-kube-api-access-g6gqf\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:00.859888 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:00.859786    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:01.476925 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:01.476889    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh" event={"ID":"d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a","Type":"ContainerDied","Data":"009b65f1417b9680c3fa66ca432ff88be79dc0b8b3037a28d2b26c5e496be386"}
Jul 08 16:42:01.477084 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:01.476957    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="009b65f1417b9680c3fa66ca432ff88be79dc0b8b3037a28d2b26c5e496be386"
Jul 08 16:42:01.477084 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:01.476903    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rvnkh"
Jul 08 16:42:04.905178 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:04.905141    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:04.999494 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:04.999469    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:10.503430 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.503398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:10.815088 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.815017    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:10.862636 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.862612    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-cfk59"]
Jul 08 16:42:10.862889 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.862877    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerName="extract"
Jul 08 16:42:10.862930 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.862891    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerName="extract"
Jul 08 16:42:10.862930 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.862902    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerName="util"
Jul 08 16:42:10.862930 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.862908    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerName="util"
Jul 08 16:42:10.862930 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.862920    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerName="pull"
Jul 08 16:42:10.862930 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.862926    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerName="pull"
Jul 08 16:42:10.863077 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.862973    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d367cb4e-fb71-4b83-9ce6-9d05fa67fd3a" containerName="extract"
Jul 08 16:42:10.865224 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.865211    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:10.868872 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.868787    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xvhp6\""
Jul 08 16:42:10.868872 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.868831    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 08 16:42:10.868872 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.868839    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 08 16:42:10.869099 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.868842    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 08 16:42:10.869099 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.868834    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 08 16:42:10.869099 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.868909    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 08 16:42:10.877456 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.877437    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-cfk59"]
Jul 08 16:42:10.890816 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.890794    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:10.929792 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.929770    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e42fccf4-f9d7-4882-8df3-cb7ea1862492-cabundle0\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:10.929890 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.929822    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-certificates\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:10.929890 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:10.929846    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtgr\" (UniqueName: \"kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-kube-api-access-pwtgr\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.031082 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.031062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-certificates\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.031177 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.031089    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtgr\" (UniqueName: \"kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-kube-api-access-pwtgr\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.031177 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.031114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e42fccf4-f9d7-4882-8df3-cb7ea1862492-cabundle0\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.031252 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:42:11.031203    2575 secret.go:281] references non-existent secret key: ca.crt
Jul 08 16:42:11.031252 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:42:11.031217    2575 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 08 16:42:11.031252 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:42:11.031225    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-cfk59: references non-existent secret key: ca.crt
Jul 08 16:42:11.031355 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:42:11.031268    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-certificates podName:e42fccf4-f9d7-4882-8df3-cb7ea1862492 nodeName:}" failed. No retries permitted until 2026-07-08 16:42:11.531255924 +0000 UTC m=+472.520812644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-certificates") pod "keda-operator-68bff59c-cfk59" (UID: "e42fccf4-f9d7-4882-8df3-cb7ea1862492") : references non-existent secret key: ca.crt
Jul 08 16:42:11.031701 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.031685    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e42fccf4-f9d7-4882-8df3-cb7ea1862492-cabundle0\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.040165 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.040147    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtgr\" (UniqueName: \"kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-kube-api-access-pwtgr\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.535499 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.535469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-certificates\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.537979 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.537956    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e42fccf4-f9d7-4882-8df3-cb7ea1862492-certificates\") pod \"keda-operator-68bff59c-cfk59\" (UID: \"e42fccf4-f9d7-4882-8df3-cb7ea1862492\") " pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.592178 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.592152    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"]
Jul 08 16:42:11.594438 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.594423    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.597167 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.597152    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 08 16:42:11.608346 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.608327    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"]
Jul 08 16:42:11.736743 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.736714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a16b09a5-dc02-404b-9805-5099a1d7a173-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.736743 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.736744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-certificates\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.736900 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.736774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsg8k\" (UniqueName: \"kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-kube-api-access-tsg8k\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.774751 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.774725    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:11.837904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.837874    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a16b09a5-dc02-404b-9805-5099a1d7a173-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.838039 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.837923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-certificates\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.838039 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.837960    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsg8k\" (UniqueName: \"kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-kube-api-access-tsg8k\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.838163 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:42:11.838105    2575 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 08 16:42:11.838163 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:42:11.838142    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr: secret "keda-metrics-apiserver-certs" not found
Jul 08 16:42:11.838267 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:42:11.838202    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-certificates podName:a16b09a5-dc02-404b-9805-5099a1d7a173 nodeName:}" failed. No retries permitted until 2026-07-08 16:42:12.338186345 +0000 UTC m=+473.327743071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-certificates") pod "keda-metrics-apiserver-559f75f947-tnmrr" (UID: "a16b09a5-dc02-404b-9805-5099a1d7a173") : secret "keda-metrics-apiserver-certs" not found
Jul 08 16:42:11.838412 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.838387    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a16b09a5-dc02-404b-9805-5099a1d7a173-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.848170 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.848147    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsg8k\" (UniqueName: \"kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-kube-api-access-tsg8k\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:11.905103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:11.905082    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-cfk59"]
Jul 08 16:42:11.907166 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:42:11.907142    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42fccf4_f9d7_4882_8df3_cb7ea1862492.slice/crio-14502b83e35258115199647a0d863943df48d367f1febd26876a727736cf405d WatchSource:0}: Error finding container 14502b83e35258115199647a0d863943df48d367f1febd26876a727736cf405d: Status 404 returned error can't find the container with id 14502b83e35258115199647a0d863943df48d367f1febd26876a727736cf405d
Jul 08 16:42:12.342616 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:12.342588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-certificates\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:12.345123 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:12.345103    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a16b09a5-dc02-404b-9805-5099a1d7a173-certificates\") pod \"keda-metrics-apiserver-559f75f947-tnmrr\" (UID: \"a16b09a5-dc02-404b-9805-5099a1d7a173\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:12.503981 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:12.503956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:12.518465 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:12.518436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-cfk59" event={"ID":"e42fccf4-f9d7-4882-8df3-cb7ea1862492","Type":"ContainerStarted","Data":"14502b83e35258115199647a0d863943df48d367f1febd26876a727736cf405d"}
Jul 08 16:42:12.630826 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:12.630803    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"]
Jul 08 16:42:12.632867 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:42:12.632836    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16b09a5_dc02_404b_9805_5099a1d7a173.slice/crio-2992f475cc1345f0eca66042345a1a9a730fd08b1723f8bbd16e3f7b24389753 WatchSource:0}: Error finding container 2992f475cc1345f0eca66042345a1a9a730fd08b1723f8bbd16e3f7b24389753: Status 404 returned error can't find the container with id 2992f475cc1345f0eca66042345a1a9a730fd08b1723f8bbd16e3f7b24389753
Jul 08 16:42:13.263992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:13.263966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:13.380757 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:13.380731    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:13.522915 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:13.522853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr" event={"ID":"a16b09a5-dc02-404b-9805-5099a1d7a173","Type":"ContainerStarted","Data":"2992f475cc1345f0eca66042345a1a9a730fd08b1723f8bbd16e3f7b24389753"}
Jul 08 16:42:17.540092 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:17.540045    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-cfk59" event={"ID":"e42fccf4-f9d7-4882-8df3-cb7ea1862492","Type":"ContainerStarted","Data":"f4b1bbaac4eda7e4fa33e306667ce232af95fc5d7fbf7eb3c24091128246de2e"}
Jul 08 16:42:17.540557 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:17.540244    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:17.541937 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:17.541911    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr" event={"ID":"a16b09a5-dc02-404b-9805-5099a1d7a173","Type":"ContainerStarted","Data":"6653fab3543df1dd52117fa58ad70f3d6b0b007ea88cc653f9d74ea279525399"}
Jul 08 16:42:17.542087 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:17.542064    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:17.558992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:17.558942    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-cfk59" podStartSLOduration=2.013014776 podStartE2EDuration="7.558924012s" totalImagesPullingTime="5.545909236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:17.557956724 +0000 UTC m=+478.547513492" watchObservedRunningTime="2026-07-08 16:42:17.558924012 +0000 UTC m=+478.548480754"
Jul 08 16:42:17.576850 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:17.576806    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr" podStartSLOduration=1.761041805 podStartE2EDuration="6.576794741s" totalImagesPullingTime="4.815752936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:17.575825825 +0000 UTC m=+478.565382598" watchObservedRunningTime="2026-07-08 16:42:17.576794741 +0000 UTC m=+478.566351485"
Jul 08 16:42:20.370222 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.370190    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64d9866d66-z6bdl"]
Jul 08 16:42:20.372300 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.372284    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.385682 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.385636    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64d9866d66-z6bdl"]
Jul 08 16:42:20.451484 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.451453    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:20.505578 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.505556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-trusted-ca-bundle\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.505698 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.505589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-service-ca\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.505698 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.505607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gq74\" (UniqueName: \"kubernetes.io/projected/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-kube-api-access-5gq74\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.505698 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.505635    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-serving-cert\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.505698 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.505670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-oauth-serving-cert\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.505858 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.505698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-oauth-config\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.505858 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.505729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-config\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.606837 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.606814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-trusted-ca-bundle\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.606943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.606859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-service-ca\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.606943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.606886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gq74\" (UniqueName: \"kubernetes.io/projected/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-kube-api-access-5gq74\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.606943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.606917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-serving-cert\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.607102 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.606939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-oauth-serving-cert\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.607102 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.606970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-oauth-config\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.607102 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.607017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-config\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.607552 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.607533    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-service-ca\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.607691 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.607575    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-oauth-serving-cert\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.607844 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.607823    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-config\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.607900 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.607880    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-trusted-ca-bundle\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.609431 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.609406    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-oauth-config\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.609629 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.609596    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-console-serving-cert\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.616405 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.616384    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gq74\" (UniqueName: \"kubernetes.io/projected/0b192d5b-324e-4a8d-8863-bb48b3d4ed63-kube-api-access-5gq74\") pod \"console-64d9866d66-z6bdl\" (UID: \"0b192d5b-324e-4a8d-8863-bb48b3d4ed63\") " pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.630266 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.630218    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:20.681166 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.681145    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:20.720974 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.720934    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:20.806375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:20.806351    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64d9866d66-z6bdl"]
Jul 08 16:42:20.808817 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:42:20.808788    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b192d5b_324e_4a8d_8863_bb48b3d4ed63.slice/crio-5605f8ab9899a98c23fb9cf3b26f5cf594adccffeae67a2d5f2e5939322b1040 WatchSource:0}: Error finding container 5605f8ab9899a98c23fb9cf3b26f5cf594adccffeae67a2d5f2e5939322b1040: Status 404 returned error can't find the container with id 5605f8ab9899a98c23fb9cf3b26f5cf594adccffeae67a2d5f2e5939322b1040
Jul 08 16:42:21.236169 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:21.236136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:21.558120 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:21.558038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d9866d66-z6bdl" event={"ID":"0b192d5b-324e-4a8d-8863-bb48b3d4ed63","Type":"ContainerStarted","Data":"eeca25119cc0d50910b5d183a543671e9844277fc4989966ac34289c605fed01"}
Jul 08 16:42:21.558120 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:21.558084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d9866d66-z6bdl" event={"ID":"0b192d5b-324e-4a8d-8863-bb48b3d4ed63","Type":"ContainerStarted","Data":"5605f8ab9899a98c23fb9cf3b26f5cf594adccffeae67a2d5f2e5939322b1040"}
Jul 08 16:42:21.577276 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:21.577231    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64d9866d66-z6bdl" podStartSLOduration=1.5772175750000001 podStartE2EDuration="1.577217575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:21.576374378 +0000 UTC m=+482.565931158" watchObservedRunningTime="2026-07-08 16:42:21.577217575 +0000 UTC m=+482.566774334"
Jul 08 16:42:23.143150 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:23.143114    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:28.550303 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:28.550276    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tnmrr"
Jul 08 16:42:30.682893 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:30.682840    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:30.683330 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:30.682997    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:30.687440 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:30.687410    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:31.600211 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:31.600188    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64d9866d66-z6bdl"
Jul 08 16:42:31.662288 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:31.662254    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69b5d56455-c252j"]
Jul 08 16:42:31.810156 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:31.810121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:31.910761 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:31.910688    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:38.548498 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:38.548472    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-cfk59"
Jul 08 16:42:56.684943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.684893    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-69b5d56455-c252j" podUID="cf041cca-8f0a-40be-88ee-634f3eaf40bf" containerName="console" containerID="cri-o://d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137" gracePeriod=15
Jul 08 16:42:56.917915 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.917893    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:42:56.963566 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.963512    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-serving-cert\") pod \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") "
Jul 08 16:42:56.963566 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.963555    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qdxb\" (UniqueName: \"kubernetes.io/projected/cf041cca-8f0a-40be-88ee-634f3eaf40bf-kube-api-access-6qdxb\") pod \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") "
Jul 08 16:42:56.963732 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.963583    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-service-ca\") pod \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") "
Jul 08 16:42:56.963732 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.963613    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-trusted-ca-bundle\") pod \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") "
Jul 08 16:42:56.963732 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.963670    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-oauth-serving-cert\") pod \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") "
Jul 08 16:42:56.963876 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.963736    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-oauth-config\") pod \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") "
Jul 08 16:42:56.963876 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.963764    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-config\") pod \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\" (UID: \"cf041cca-8f0a-40be-88ee-634f3eaf40bf\") "
Jul 08 16:42:56.964121 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.964087    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-service-ca" (OuterVolumeSpecName: "service-ca") pod "cf041cca-8f0a-40be-88ee-634f3eaf40bf" (UID: "cf041cca-8f0a-40be-88ee-634f3eaf40bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:42:56.964205 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.964110    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cf041cca-8f0a-40be-88ee-634f3eaf40bf" (UID: "cf041cca-8f0a-40be-88ee-634f3eaf40bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:42:56.964205 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.964095    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cf041cca-8f0a-40be-88ee-634f3eaf40bf" (UID: "cf041cca-8f0a-40be-88ee-634f3eaf40bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:42:56.964325 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.964299    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-config" (OuterVolumeSpecName: "console-config") pod "cf041cca-8f0a-40be-88ee-634f3eaf40bf" (UID: "cf041cca-8f0a-40be-88ee-634f3eaf40bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:42:56.965669 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.965613    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cf041cca-8f0a-40be-88ee-634f3eaf40bf" (UID: "cf041cca-8f0a-40be-88ee-634f3eaf40bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:42:56.965769 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.965702    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf041cca-8f0a-40be-88ee-634f3eaf40bf-kube-api-access-6qdxb" (OuterVolumeSpecName: "kube-api-access-6qdxb") pod "cf041cca-8f0a-40be-88ee-634f3eaf40bf" (UID: "cf041cca-8f0a-40be-88ee-634f3eaf40bf"). InnerVolumeSpecName "kube-api-access-6qdxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:42:56.965846 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:56.965783    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cf041cca-8f0a-40be-88ee-634f3eaf40bf" (UID: "cf041cca-8f0a-40be-88ee-634f3eaf40bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:42:57.064511 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.064485    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qdxb\" (UniqueName: \"kubernetes.io/projected/cf041cca-8f0a-40be-88ee-634f3eaf40bf-kube-api-access-6qdxb\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:57.064511 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.064510    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:57.064714 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.064525    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-trusted-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:57.064714 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.064537    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-oauth-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:57.064714 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.064549    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-oauth-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:57.064714 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.064562    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:57.064714 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.064575    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf041cca-8f0a-40be-88ee-634f3eaf40bf-console-serving-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:42:57.693073 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.693040    2575 generic.go:356] "Generic (PLEG): container finished" podID="cf041cca-8f0a-40be-88ee-634f3eaf40bf" containerID="d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137" exitCode=0
Jul 08 16:42:57.693413 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.693085    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b5d56455-c252j"
Jul 08 16:42:57.693413 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.693112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b5d56455-c252j" event={"ID":"cf041cca-8f0a-40be-88ee-634f3eaf40bf","Type":"ContainerDied","Data":"d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137"}
Jul 08 16:42:57.693413 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.693153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b5d56455-c252j" event={"ID":"cf041cca-8f0a-40be-88ee-634f3eaf40bf","Type":"ContainerDied","Data":"a584ed129edfee5755cf30566bf7bc69a5151aa045939c476c046cbd94d5ce13"}
Jul 08 16:42:57.693413 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.693181    2575 scope.go:117] "RemoveContainer" containerID="d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137"
Jul 08 16:42:57.700978 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.700960    2575 scope.go:117] "RemoveContainer" containerID="d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137"
Jul 08 16:42:57.701223 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:42:57.701202    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137\": container with ID starting with d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137 not found: ID does not exist" containerID="d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137"
Jul 08 16:42:57.701287 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.701230    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137"} err="failed to get container status \"d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137\": rpc error: code = NotFound desc = could not find container \"d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137\": container with ID starting with d0ff96bc42adc1c11a25b295895ac461da293c3d1bcf6c7542cf6a1023681137 not found: ID does not exist"
Jul 08 16:42:57.709303 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.709283    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69b5d56455-c252j"]
Jul 08 16:42:57.712733 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:57.712715    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69b5d56455-c252j"]
Jul 08 16:42:59.607736 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:42:59.607702    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf041cca-8f0a-40be-88ee-634f3eaf40bf" path="/var/lib/kubelet/pods/cf041cca-8f0a-40be-88ee-634f3eaf40bf/volumes"
Jul 08 16:43:15.085001 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:15.084911    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:15.174093 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:15.174058    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:20.442845 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.442810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:20.545883 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.545845    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:20.621567 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.621540    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-rnhkw"]
Jul 08 16:43:20.621821 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.621808    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf041cca-8f0a-40be-88ee-634f3eaf40bf" containerName="console"
Jul 08 16:43:20.621865 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.621823    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf041cca-8f0a-40be-88ee-634f3eaf40bf" containerName="console"
Jul 08 16:43:20.621898 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.621882    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf041cca-8f0a-40be-88ee-634f3eaf40bf" containerName="console"
Jul 08 16:43:20.624369 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.624349    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:20.627561 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.627541    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mcq4x\""
Jul 08 16:43:20.627633 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.627603    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:43:20.627633 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.627619    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 16:43:20.628443 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.628428    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:43:20.636067 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.636048    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rnhkw"]
Jul 08 16:43:20.677566 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.677537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:20.724482 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.724431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e86c348f-0af6-40dc-8789-cee488757371-data\") pod \"seaweedfs-64568bcd49-rnhkw\" (UID: \"e86c348f-0af6-40dc-8789-cee488757371\") " pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:20.724557 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.724475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qbrx\" (UniqueName: \"kubernetes.io/projected/e86c348f-0af6-40dc-8789-cee488757371-kube-api-access-6qbrx\") pod \"seaweedfs-64568bcd49-rnhkw\" (UID: \"e86c348f-0af6-40dc-8789-cee488757371\") " pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:20.783790 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.783764    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:20.824856 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.824837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e86c348f-0af6-40dc-8789-cee488757371-data\") pod \"seaweedfs-64568bcd49-rnhkw\" (UID: \"e86c348f-0af6-40dc-8789-cee488757371\") " pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:20.824931 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.824866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qbrx\" (UniqueName: \"kubernetes.io/projected/e86c348f-0af6-40dc-8789-cee488757371-kube-api-access-6qbrx\") pod \"seaweedfs-64568bcd49-rnhkw\" (UID: \"e86c348f-0af6-40dc-8789-cee488757371\") " pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:20.825191 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.825176    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e86c348f-0af6-40dc-8789-cee488757371-data\") pod \"seaweedfs-64568bcd49-rnhkw\" (UID: \"e86c348f-0af6-40dc-8789-cee488757371\") " pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:20.838176 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.838152    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qbrx\" (UniqueName: \"kubernetes.io/projected/e86c348f-0af6-40dc-8789-cee488757371-kube-api-access-6qbrx\") pod \"seaweedfs-64568bcd49-rnhkw\" (UID: \"e86c348f-0af6-40dc-8789-cee488757371\") " pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:20.885702 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.885679    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:20.933558 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:20.933537    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:21.059821 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:21.059798    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rnhkw"]
Jul 08 16:43:21.062085 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:43:21.062056    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86c348f_0af6_40dc_8789_cee488757371.slice/crio-b476ca346ff31df28789f3bec7c88fa3fef0f533cc8f7a682aab18ff5552ea6f WatchSource:0}: Error finding container b476ca346ff31df28789f3bec7c88fa3fef0f533cc8f7a682aab18ff5552ea6f: Status 404 returned error can't find the container with id b476ca346ff31df28789f3bec7c88fa3fef0f533cc8f7a682aab18ff5552ea6f
Jul 08 16:43:21.197891 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:21.197861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:21.297565 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:21.297499    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:21.783728 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:21.783677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rnhkw" event={"ID":"e86c348f-0af6-40dc-8789-cee488757371","Type":"ContainerStarted","Data":"b476ca346ff31df28789f3bec7c88fa3fef0f533cc8f7a682aab18ff5552ea6f"}
Jul 08 16:43:26.806073 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:26.806038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rnhkw" event={"ID":"e86c348f-0af6-40dc-8789-cee488757371","Type":"ContainerStarted","Data":"58ff2a2abfad6582fb04d7113ae781cbb99f71405056684385be573f070ca1b8"}
Jul 08 16:43:26.806456 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:26.806152    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:26.821571 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:26.821532    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-rnhkw" podStartSLOduration=1.810374544 podStartE2EDuration="6.821520166s" totalImagesPullingTime="5.011145622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:26.820975608 +0000 UTC m=+547.810532355" watchObservedRunningTime="2026-07-08 16:43:26.821520166 +0000 UTC m=+547.811076908"
Jul 08 16:43:32.811167 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:32.811135    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-rnhkw"
Jul 08 16:43:50.032506 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:50.032457    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:58.345051 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:43:58.345013    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:19.447958 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:19.447932    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:44:19.453484 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:19.453463    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:44:29.281916 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.281878    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:29.325126 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.325101    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-fntrg"]
Jul 08 16:44:29.328161 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.328141    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:29.330606 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.330585    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 08 16:44:29.330817 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.330798    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tzw8m\""
Jul 08 16:44:29.338175 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.338154    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-fntrg"]
Jul 08 16:44:29.345617 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.345595    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xnzct"]
Jul 08 16:44:29.348696 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.348673    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:29.350818 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.350748    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:29.350935 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.350899    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 08 16:44:29.351004 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.350944    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-t2rnx\""
Jul 08 16:44:29.359902 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.359883    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xnzct"]
Jul 08 16:44:29.403489 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.403470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b46c0ef-bf63-4cf7-a413-11e6351a4def-tls-certs\") pod \"model-serving-api-fd65d7d6-fntrg\" (UID: \"0b46c0ef-bf63-4cf7-a413-11e6351a4def\") " pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:29.403578 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.403496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d93a62a9-b35c-47aa-a075-d85d06f31068-cert\") pod \"odh-model-controller-84b54b7f97-xnzct\" (UID: \"d93a62a9-b35c-47aa-a075-d85d06f31068\") " pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:29.403578 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.403525    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxxl\" (UniqueName: \"kubernetes.io/projected/0b46c0ef-bf63-4cf7-a413-11e6351a4def-kube-api-access-sfxxl\") pod \"model-serving-api-fd65d7d6-fntrg\" (UID: \"0b46c0ef-bf63-4cf7-a413-11e6351a4def\") " pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:29.403669 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.403591    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbmm\" (UniqueName: \"kubernetes.io/projected/d93a62a9-b35c-47aa-a075-d85d06f31068-kube-api-access-vlbmm\") pod \"odh-model-controller-84b54b7f97-xnzct\" (UID: \"d93a62a9-b35c-47aa-a075-d85d06f31068\") " pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:29.504294 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.504272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxxl\" (UniqueName: \"kubernetes.io/projected/0b46c0ef-bf63-4cf7-a413-11e6351a4def-kube-api-access-sfxxl\") pod \"model-serving-api-fd65d7d6-fntrg\" (UID: \"0b46c0ef-bf63-4cf7-a413-11e6351a4def\") " pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:29.504355 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.504318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbmm\" (UniqueName: \"kubernetes.io/projected/d93a62a9-b35c-47aa-a075-d85d06f31068-kube-api-access-vlbmm\") pod \"odh-model-controller-84b54b7f97-xnzct\" (UID: \"d93a62a9-b35c-47aa-a075-d85d06f31068\") " pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:29.504391 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.504355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b46c0ef-bf63-4cf7-a413-11e6351a4def-tls-certs\") pod \"model-serving-api-fd65d7d6-fntrg\" (UID: \"0b46c0ef-bf63-4cf7-a413-11e6351a4def\") " pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:29.504391 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.504369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d93a62a9-b35c-47aa-a075-d85d06f31068-cert\") pod \"odh-model-controller-84b54b7f97-xnzct\" (UID: \"d93a62a9-b35c-47aa-a075-d85d06f31068\") " pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:29.504489 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:44:29.504473    2575 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 08 16:44:29.504489 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:44:29.504483    2575 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 08 16:44:29.504560 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:44:29.504536    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b46c0ef-bf63-4cf7-a413-11e6351a4def-tls-certs podName:0b46c0ef-bf63-4cf7-a413-11e6351a4def nodeName:}" failed. No retries permitted until 2026-07-08 16:44:30.004518321 +0000 UTC m=+610.994075053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b46c0ef-bf63-4cf7-a413-11e6351a4def-tls-certs") pod "model-serving-api-fd65d7d6-fntrg" (UID: "0b46c0ef-bf63-4cf7-a413-11e6351a4def") : secret "model-serving-api-tls" not found
Jul 08 16:44:29.504560 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:44:29.504550    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93a62a9-b35c-47aa-a075-d85d06f31068-cert podName:d93a62a9-b35c-47aa-a075-d85d06f31068 nodeName:}" failed. No retries permitted until 2026-07-08 16:44:30.004543669 +0000 UTC m=+610.994100390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d93a62a9-b35c-47aa-a075-d85d06f31068-cert") pod "odh-model-controller-84b54b7f97-xnzct" (UID: "d93a62a9-b35c-47aa-a075-d85d06f31068") : secret "odh-model-controller-webhook-cert" not found
Jul 08 16:44:29.515478 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.515458    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbmm\" (UniqueName: \"kubernetes.io/projected/d93a62a9-b35c-47aa-a075-d85d06f31068-kube-api-access-vlbmm\") pod \"odh-model-controller-84b54b7f97-xnzct\" (UID: \"d93a62a9-b35c-47aa-a075-d85d06f31068\") " pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:29.515543 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.515480    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxxl\" (UniqueName: \"kubernetes.io/projected/0b46c0ef-bf63-4cf7-a413-11e6351a4def-kube-api-access-sfxxl\") pod \"model-serving-api-fd65d7d6-fntrg\" (UID: \"0b46c0ef-bf63-4cf7-a413-11e6351a4def\") " pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:29.594743 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.594689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:29.696898 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:29.696860    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:30.008599 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.008555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b46c0ef-bf63-4cf7-a413-11e6351a4def-tls-certs\") pod \"model-serving-api-fd65d7d6-fntrg\" (UID: \"0b46c0ef-bf63-4cf7-a413-11e6351a4def\") " pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:30.008599 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.008600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d93a62a9-b35c-47aa-a075-d85d06f31068-cert\") pod \"odh-model-controller-84b54b7f97-xnzct\" (UID: \"d93a62a9-b35c-47aa-a075-d85d06f31068\") " pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:30.010925 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.010897    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b46c0ef-bf63-4cf7-a413-11e6351a4def-tls-certs\") pod \"model-serving-api-fd65d7d6-fntrg\" (UID: \"0b46c0ef-bf63-4cf7-a413-11e6351a4def\") " pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:30.011079 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.011056    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d93a62a9-b35c-47aa-a075-d85d06f31068-cert\") pod \"odh-model-controller-84b54b7f97-xnzct\" (UID: \"d93a62a9-b35c-47aa-a075-d85d06f31068\") " pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:30.084493 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.084466    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:30.148994 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.148959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:30.238818 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.238772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:30.257944 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.257918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:30.371937 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.371910    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-fntrg"]
Jul 08 16:44:30.373900 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:44:30.373857    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b46c0ef_bf63_4cf7_a413_11e6351a4def.slice/crio-cd753aa782573ffe835ab5d9ec5f340ce19341bbc35039d0dbc147a795628ac6 WatchSource:0}: Error finding container cd753aa782573ffe835ab5d9ec5f340ce19341bbc35039d0dbc147a795628ac6: Status 404 returned error can't find the container with id cd753aa782573ffe835ab5d9ec5f340ce19341bbc35039d0dbc147a795628ac6
Jul 08 16:44:30.395508 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:30.395488    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xnzct"]
Jul 08 16:44:30.396896 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:44:30.396870    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93a62a9_b35c_47aa_a075_d85d06f31068.slice/crio-08050a1b2bad29a5f9d4eaf4cfb9a1ff0572ce21f822a0883acd63fbac188794 WatchSource:0}: Error finding container 08050a1b2bad29a5f9d4eaf4cfb9a1ff0572ce21f822a0883acd63fbac188794: Status 404 returned error can't find the container with id 08050a1b2bad29a5f9d4eaf4cfb9a1ff0572ce21f822a0883acd63fbac188794
Jul 08 16:44:31.043750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:31.043660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-fntrg" event={"ID":"0b46c0ef-bf63-4cf7-a413-11e6351a4def","Type":"ContainerStarted","Data":"cd753aa782573ffe835ab5d9ec5f340ce19341bbc35039d0dbc147a795628ac6"}
Jul 08 16:44:31.045798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:31.045734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-xnzct" event={"ID":"d93a62a9-b35c-47aa-a075-d85d06f31068","Type":"ContainerStarted","Data":"08050a1b2bad29a5f9d4eaf4cfb9a1ff0572ce21f822a0883acd63fbac188794"}
Jul 08 16:44:34.059304 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:34.059273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-fntrg" event={"ID":"0b46c0ef-bf63-4cf7-a413-11e6351a4def","Type":"ContainerStarted","Data":"dfad6c3dc7db56fe64804e614259fa910fbd5a53336ddf18219c1add73655492"}
Jul 08 16:44:34.059711 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:34.059405    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:44:34.060758 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:34.060736    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-xnzct" event={"ID":"d93a62a9-b35c-47aa-a075-d85d06f31068","Type":"ContainerStarted","Data":"c034002f23dd1a7ed7e7db4d1067e48d6328e0a2a6a075da732155a054fa9df8"}
Jul 08 16:44:34.060846 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:34.060785    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:34.121057 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:34.121012    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-fntrg" podStartSLOduration=1.566629717 podStartE2EDuration="5.120996602s" totalImagesPullingTime="3.554366885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:44:34.107194023 +0000 UTC m=+615.096750779" watchObservedRunningTime="2026-07-08 16:44:34.120996602 +0000 UTC m=+615.110553346"
Jul 08 16:44:34.187587 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:34.187503    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-xnzct" podStartSLOduration=1.706725751 podStartE2EDuration="5.187485194s" totalImagesPullingTime="3.480759443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:44:34.185743572 +0000 UTC m=+615.175300318" watchObservedRunningTime="2026-07-08 16:44:34.187485194 +0000 UTC m=+615.177041937"
Jul 08 16:44:35.084997 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:35.084957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:45.066110 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:45.066036    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-xnzct"
Jul 08 16:44:45.068022 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:44:45.068001    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-fntrg"
Jul 08 16:45:27.970001 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:45:27.969959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:05.271760 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:05.271726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:05.393805 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:05.393767    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:09.373038 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:09.373001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:17.795009 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:17.794933    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:17.897688 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:17.897628    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:18.034126 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.034098    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:18.101674 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.101633    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"]
Jul 08 16:46:18.104853 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.104835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.107879 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.107859    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:46:18.107992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.107943    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f6c4d-predictor-serving-cert\""
Jul 08 16:46:18.107992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.107952    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f6c4d-kube-rbac-proxy-sar-config\""
Jul 08 16:46:18.108109 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.108024    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:46:18.108482 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.108467    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gs76g\""
Jul 08 16:46:18.114188 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.114160    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"]
Jul 08 16:46:18.161247 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.161222    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.161341 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.161255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0161d25-bd08-40da-84ff-a62b4758a1ad-proxy-tls\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.161341 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.161329    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnf8h\" (UniqueName: \"kubernetes.io/projected/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-api-access-gnf8h\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.180080 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.180038    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:18.262154 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.262128    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnf8h\" (UniqueName: \"kubernetes.io/projected/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-api-access-gnf8h\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.262269 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.262169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.262269 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.262207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0161d25-bd08-40da-84ff-a62b4758a1ad-proxy-tls\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.262874 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.262852    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.264584 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.264566    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0161d25-bd08-40da-84ff-a62b4758a1ad-proxy-tls\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.270026 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.270000    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnf8h\" (UniqueName: \"kubernetes.io/projected/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-api-access-gnf8h\") pod \"success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") " pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.282277 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.282253    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:18.301146 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.301119    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"]
Jul 08 16:46:18.304302 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.304286    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.306931 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.306907    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f6c4d-kube-rbac-proxy-sar-config\""
Jul 08 16:46:18.307029 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.306910    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f6c4d-predictor-serving-cert\""
Jul 08 16:46:18.313753 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.313733    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"]
Jul 08 16:46:18.362684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.362602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxkv5\" (UniqueName: \"kubernetes.io/projected/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-api-access-jxkv5\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.362684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.362669    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.362845 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.362705    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-proxy-tls\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.382794 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.382739    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:18.416742 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.416720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:18.463136 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.463105    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.463296 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.463188    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-proxy-tls\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.463296 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.463245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkv5\" (UniqueName: \"kubernetes.io/projected/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-api-access-jxkv5\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.464053 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.464025    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.466189 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.466145    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-proxy-tls\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.471640 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.471619    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkv5\" (UniqueName: \"kubernetes.io/projected/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-api-access-jxkv5\") pod \"error-404-isvc-f6c4d-predictor-68f94b898b-49jc7\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") " pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.482778 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.482755    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:18.550395 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.550362    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"]
Jul 08 16:46:18.553195 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:46:18.553170    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0161d25_bd08_40da_84ff_a62b4758a1ad.slice/crio-5d1a89ebbff60412bb09f6bed647c58253fd539154bbd77f1752b9e60a15e176 WatchSource:0}: Error finding container 5d1a89ebbff60412bb09f6bed647c58253fd539154bbd77f1752b9e60a15e176: Status 404 returned error can't find the container with id 5d1a89ebbff60412bb09f6bed647c58253fd539154bbd77f1752b9e60a15e176
Jul 08 16:46:18.614945 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.614894    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:18.651404 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.651378    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:18.740867 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:18.740774    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"]
Jul 08 16:46:18.742795 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:46:18.742766    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356dfcc5_b1f8_4b29_ac1b_d9525b41c2b8.slice/crio-d1ed9fb9a21d06ea62778934fa5bd9af6fd05b71262971ac13a530380dd16762 WatchSource:0}: Error finding container d1ed9fb9a21d06ea62778934fa5bd9af6fd05b71262971ac13a530380dd16762: Status 404 returned error can't find the container with id d1ed9fb9a21d06ea62778934fa5bd9af6fd05b71262971ac13a530380dd16762
Jul 08 16:46:19.459736 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:19.459623    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" event={"ID":"e0161d25-bd08-40da-84ff-a62b4758a1ad","Type":"ContainerStarted","Data":"5d1a89ebbff60412bb09f6bed647c58253fd539154bbd77f1752b9e60a15e176"}
Jul 08 16:46:19.464134 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:19.464051    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" event={"ID":"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8","Type":"ContainerStarted","Data":"d1ed9fb9a21d06ea62778934fa5bd9af6fd05b71262971ac13a530380dd16762"}
Jul 08 16:46:32.285702 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:32.285634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:32.351361 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:32.351321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:34.555437 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:34.555412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" event={"ID":"e0161d25-bd08-40da-84ff-a62b4758a1ad","Type":"ContainerStarted","Data":"84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98"}
Jul 08 16:46:34.557279 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:34.557258    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" event={"ID":"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8","Type":"ContainerStarted","Data":"8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d"}
Jul 08 16:46:37.577144 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.577085    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" event={"ID":"e0161d25-bd08-40da-84ff-a62b4758a1ad","Type":"ContainerStarted","Data":"16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690"}
Jul 08 16:46:37.577607 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.577197    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:37.577607 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.577234    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:37.578631 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.578594    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:46:37.579615 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.579585    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" event={"ID":"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8","Type":"ContainerStarted","Data":"8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4"}
Jul 08 16:46:37.579760 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.579672    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:37.579760 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.579707    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:37.581183 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.581153    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:46:37.597857 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.597812    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podStartSLOduration=1.143292774 podStartE2EDuration="19.597799773s" totalImagesPullingTime="18.454506999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:37.596358943 +0000 UTC m=+738.585915688" watchObservedRunningTime="2026-07-08 16:46:37.597799773 +0000 UTC m=+738.587356515"
Jul 08 16:46:37.614715 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:37.614679    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podStartSLOduration=1.3711557970000001 podStartE2EDuration="19.614639969s" totalImagesPullingTime="18.243484172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:37.613553772 +0000 UTC m=+738.603110539" watchObservedRunningTime="2026-07-08 16:46:37.614639969 +0000 UTC m=+738.604196711"
Jul 08 16:46:38.584817 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:38.584754    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:46:38.585253 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:38.585107    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:46:43.590773 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:43.590741    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:46:43.591413 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:43.591385    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:46:43.591663 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:43.591620    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:46:43.591766 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:43.591672    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:46:47.413101 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:47.413068    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:53.591817 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:53.591774    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:46:53.592358 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:46:53.591774    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:47:03.592263 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:03.592228    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:47:03.592725 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:03.592222    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:47:13.592521 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:13.592483    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:47:13.593220 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:13.592494    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:47:23.592861 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:23.592827    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:47:23.593293 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:23.592920    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:47:37.818394 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:37.818360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:37.952023 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:37.951989    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:47.991235 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:47.991159    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:48.088314 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.088280    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"]
Jul 08 16:47:48.088556 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.088505    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container" containerID="cri-o://84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98" gracePeriod=30
Jul 08 16:47:48.088684 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.088558    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kube-rbac-proxy" containerID="cri-o://16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690" gracePeriod=30
Jul 08 16:47:48.119116 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.119062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:48.143530 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.143504    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"]
Jul 08 16:47:48.143810 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.143785    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container" containerID="cri-o://8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d" gracePeriod=30
Jul 08 16:47:48.143914 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.143822    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kube-rbac-proxy" containerID="cri-o://8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4" gracePeriod=30
Jul 08 16:47:48.172975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.172951    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"]
Jul 08 16:47:48.175337 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.175321    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.177922 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.177895    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-429af-kube-rbac-proxy-sar-config\""
Jul 08 16:47:48.178045 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.178029    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-429af-predictor-serving-cert\""
Jul 08 16:47:48.185560 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.185541    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"]
Jul 08 16:47:48.191368 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.191340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:48.231872 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.231849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.231999 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.231915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.231999 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.231937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxp6q\" (UniqueName: \"kubernetes.io/projected/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-api-access-gxp6q\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.263785 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.263762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:48.264568 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.264547    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"]
Jul 08 16:47:48.266906 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.266888    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.269121 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.269100    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-429af-predictor-serving-cert\""
Jul 08 16:47:48.269206 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.269138    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-429af-kube-rbac-proxy-sar-config\""
Jul 08 16:47:48.277636 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.277616    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"]
Jul 08 16:47:48.332635 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.332613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/678b5684-6ab2-4465-9ac7-78f25661f96c-proxy-tls\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.332761 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.332657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbwm\" (UniqueName: \"kubernetes.io/projected/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-api-access-msbwm\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.332761 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.332684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.332875 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:47:48.332765    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-429af-predictor-serving-cert: secret "success-200-isvc-429af-predictor-serving-cert" not found
Jul 08 16:47:48.332875 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:47:48.332832    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls podName:8ddf3e70-7c90-42fa-9d62-2037f9ec0244 nodeName:}" failed. No retries permitted until 2026-07-08 16:47:48.832814845 +0000 UTC m=+809.822371579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls") pod "success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" (UID: "8ddf3e70-7c90-42fa-9d62-2037f9ec0244") : secret "success-200-isvc-429af-predictor-serving-cert" not found
Jul 08 16:47:48.332875 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.332857    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.333012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.332879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxp6q\" (UniqueName: \"kubernetes.io/projected/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-api-access-gxp6q\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.333012 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.332905    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.333462 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.333441    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.340898 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.340876    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxp6q\" (UniqueName: \"kubernetes.io/projected/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-api-access-gxp6q\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.428866 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.428833    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:48.433991 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.433968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.434069 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.434030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/678b5684-6ab2-4465-9ac7-78f25661f96c-proxy-tls\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.434069 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.434062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbwm\" (UniqueName: \"kubernetes.io/projected/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-api-access-msbwm\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.434668 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.434636    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.436258 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.436239    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/678b5684-6ab2-4465-9ac7-78f25661f96c-proxy-tls\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.442762 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.442738    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbwm\" (UniqueName: \"kubernetes.io/projected/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-api-access-msbwm\") pod \"error-404-isvc-429af-predictor-5679c5df6-n85zs\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") " pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.576532 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.576471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.585766 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.585742    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 08 16:47:48.585862 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.585757    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 16:47:48.692120 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.692084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:48.705476 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.705454    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"]
Jul 08 16:47:48.707509 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:47:48.707486    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678b5684_6ab2_4465_9ac7_78f25661f96c.slice/crio-515f22a120f9d2a9bbea92d656204d24bf314cfc37658d1152bad92723156161 WatchSource:0}: Error finding container 515f22a120f9d2a9bbea92d656204d24bf314cfc37658d1152bad92723156161: Status 404 returned error can't find the container with id 515f22a120f9d2a9bbea92d656204d24bf314cfc37658d1152bad92723156161
Jul 08 16:47:48.837845 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.837794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.839770 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.839754    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls\") pod \"success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") " pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:48.858436 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.858407    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" event={"ID":"678b5684-6ab2-4465-9ac7-78f25661f96c","Type":"ContainerStarted","Data":"59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e"}
Jul 08 16:47:48.858555 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.858467    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" event={"ID":"678b5684-6ab2-4465-9ac7-78f25661f96c","Type":"ContainerStarted","Data":"cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10"}
Jul 08 16:47:48.858555 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.858508    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" event={"ID":"678b5684-6ab2-4465-9ac7-78f25661f96c","Type":"ContainerStarted","Data":"515f22a120f9d2a9bbea92d656204d24bf314cfc37658d1152bad92723156161"}
Jul 08 16:47:48.858555 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.858542    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:48.860278 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.860260    2575 generic.go:356] "Generic (PLEG): container finished" podID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerID="16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690" exitCode=2
Jul 08 16:47:48.860361 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.860328    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" event={"ID":"e0161d25-bd08-40da-84ff-a62b4758a1ad","Type":"ContainerDied","Data":"16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690"}
Jul 08 16:47:48.861983 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.861960    2575 generic.go:356] "Generic (PLEG): container finished" podID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerID="8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4" exitCode=2
Jul 08 16:47:48.862068 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.861981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" event={"ID":"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8","Type":"ContainerDied","Data":"8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4"}
Jul 08 16:47:48.879774 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:48.879724    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podStartSLOduration=0.879706264 podStartE2EDuration="879.706264ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:47:48.877929704 +0000 UTC m=+809.867486449" watchObservedRunningTime="2026-07-08 16:47:48.879706264 +0000 UTC m=+809.869263020"
Jul 08 16:47:49.086003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:49.085967    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:49.215194 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:49.215171    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"]
Jul 08 16:47:49.217259 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:47:49.217231    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddf3e70_7c90_42fa_9d62_2037f9ec0244.slice/crio-e674c92e434592e2fc16bd50a76c8adabc42ae49cb12a2104f9b66a972d3f434 WatchSource:0}: Error finding container e674c92e434592e2fc16bd50a76c8adabc42ae49cb12a2104f9b66a972d3f434: Status 404 returned error can't find the container with id e674c92e434592e2fc16bd50a76c8adabc42ae49cb12a2104f9b66a972d3f434
Jul 08 16:47:49.867311 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:49.867277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" event={"ID":"8ddf3e70-7c90-42fa-9d62-2037f9ec0244","Type":"ContainerStarted","Data":"7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a"}
Jul 08 16:47:49.867511 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:49.867444    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:49.867511 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:49.867498    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" event={"ID":"8ddf3e70-7c90-42fa-9d62-2037f9ec0244","Type":"ContainerStarted","Data":"5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35"}
Jul 08 16:47:49.867615 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:49.867534    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" event={"ID":"8ddf3e70-7c90-42fa-9d62-2037f9ec0244","Type":"ContainerStarted","Data":"e674c92e434592e2fc16bd50a76c8adabc42ae49cb12a2104f9b66a972d3f434"}
Jul 08 16:47:49.868992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:49.868964    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 16:47:49.886424 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:49.886380    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podStartSLOduration=1.886364604 podStartE2EDuration="1.886364604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:47:49.885369613 +0000 UTC m=+810.874926366" watchObservedRunningTime="2026-07-08 16:47:49.886364604 +0000 UTC m=+810.875921346"
Jul 08 16:47:50.871709 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:50.871670    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:50.872148 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:50.871737    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:50.872148 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:50.871754    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 16:47:50.873063 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:50.873040    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:47:51.228720 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.228694    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:47:51.231629 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.231607    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:47:51.257438 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.257416    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnf8h\" (UniqueName: \"kubernetes.io/projected/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-api-access-gnf8h\") pod \"e0161d25-bd08-40da-84ff-a62b4758a1ad\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") "
Jul 08 16:47:51.257550 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.257477    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-proxy-tls\") pod \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") "
Jul 08 16:47:51.257550 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.257494    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0161d25-bd08-40da-84ff-a62b4758a1ad-proxy-tls\") pod \"e0161d25-bd08-40da-84ff-a62b4758a1ad\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") "
Jul 08 16:47:51.257550 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.257519    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-rbac-proxy-sar-config\") pod \"e0161d25-bd08-40da-84ff-a62b4758a1ad\" (UID: \"e0161d25-bd08-40da-84ff-a62b4758a1ad\") "
Jul 08 16:47:51.257550 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.257537    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxkv5\" (UniqueName: \"kubernetes.io/projected/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-api-access-jxkv5\") pod \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") "
Jul 08 16:47:51.258122 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.257827    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-rbac-proxy-sar-config\") pod \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\" (UID: \"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8\") "
Jul 08 16:47:51.258230 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.258169    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" (UID: "356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:47:51.258483 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.258454    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0161d25-bd08-40da-84ff-a62b4758a1ad" (UID: "e0161d25-bd08-40da-84ff-a62b4758a1ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:47:51.259807 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.259776    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-api-access-jxkv5" (OuterVolumeSpecName: "kube-api-access-jxkv5") pod "356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" (UID: "356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8"). InnerVolumeSpecName "kube-api-access-jxkv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:47:51.259978 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.259946    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-api-access-gnf8h" (OuterVolumeSpecName: "kube-api-access-gnf8h") pod "e0161d25-bd08-40da-84ff-a62b4758a1ad" (UID: "e0161d25-bd08-40da-84ff-a62b4758a1ad"). InnerVolumeSpecName "kube-api-access-gnf8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:47:51.260083 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.260043    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0161d25-bd08-40da-84ff-a62b4758a1ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0161d25-bd08-40da-84ff-a62b4758a1ad" (UID: "e0161d25-bd08-40da-84ff-a62b4758a1ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:47:51.260186 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.260165    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" (UID: "356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:47:51.358664 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.358619    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:47:51.358664 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.358641    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0161d25-bd08-40da-84ff-a62b4758a1ad-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:47:51.358664 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.358668    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:47:51.358850 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.358677    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxkv5\" (UniqueName: \"kubernetes.io/projected/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-api-access-jxkv5\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:47:51.358850 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.358686    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:47:51.358850 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.358695    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnf8h\" (UniqueName: \"kubernetes.io/projected/e0161d25-bd08-40da-84ff-a62b4758a1ad-kube-api-access-gnf8h\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:47:51.878402 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.878324    2575 generic.go:356] "Generic (PLEG): container finished" podID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerID="84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98" exitCode=0
Jul 08 16:47:51.878402 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.878390    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"
Jul 08 16:47:51.878850 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.878410    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" event={"ID":"e0161d25-bd08-40da-84ff-a62b4758a1ad","Type":"ContainerDied","Data":"84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98"}
Jul 08 16:47:51.878850 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.878486    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t" event={"ID":"e0161d25-bd08-40da-84ff-a62b4758a1ad","Type":"ContainerDied","Data":"5d1a89ebbff60412bb09f6bed647c58253fd539154bbd77f1752b9e60a15e176"}
Jul 08 16:47:51.878850 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.878527    2575 scope.go:117] "RemoveContainer" containerID="16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690"
Jul 08 16:47:51.880378 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.880357    2575 generic.go:356] "Generic (PLEG): container finished" podID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerID="8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d" exitCode=0
Jul 08 16:47:51.881155 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.881105    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"
Jul 08 16:47:51.881382 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.881359    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" event={"ID":"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8","Type":"ContainerDied","Data":"8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d"}
Jul 08 16:47:51.881473 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.881420    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7" event={"ID":"356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8","Type":"ContainerDied","Data":"d1ed9fb9a21d06ea62778934fa5bd9af6fd05b71262971ac13a530380dd16762"}
Jul 08 16:47:51.881803 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.881771    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:47:51.888335 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.888318    2575 scope.go:117] "RemoveContainer" containerID="84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98"
Jul 08 16:47:51.895288 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.895272    2575 scope.go:117] "RemoveContainer" containerID="16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690"
Jul 08 16:47:51.902087 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:47:51.898135    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690\": container with ID starting with 16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690 not found: ID does not exist" containerID="16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690"
Jul 08 16:47:51.902087 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.898173    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690"} err="failed to get container status \"16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690\": rpc error: code = NotFound desc = could not find container \"16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690\": container with ID starting with 16fe97f0299bd2a99ed981cb055d80068908f5308fd94fe2b916739dad810690 not found: ID does not exist"
Jul 08 16:47:51.902087 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.898206    2575 scope.go:117] "RemoveContainer" containerID="84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98"
Jul 08 16:47:51.902087 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:47:51.899958    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98\": container with ID starting with 84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98 not found: ID does not exist" containerID="84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98"
Jul 08 16:47:51.902087 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.899993    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98"} err="failed to get container status \"84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98\": rpc error: code = NotFound desc = could not find container \"84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98\": container with ID starting with 84657cb73495419d502f1217635d4f9923bcd9d9f7605f135243bc87c09e3e98 not found: ID does not exist"
Jul 08 16:47:51.902087 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.900068    2575 scope.go:117] "RemoveContainer" containerID="8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4"
Jul 08 16:47:51.902983 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.902964    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"]
Jul 08 16:47:51.904475 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.904454    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f6c4d-predictor-58f8f4fd9f-bbf8t"]
Jul 08 16:47:51.908685 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.908634    2575 scope.go:117] "RemoveContainer" containerID="8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d"
Jul 08 16:47:51.913909 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.913885    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"]
Jul 08 16:47:51.918160 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.918140    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6c4d-predictor-68f94b898b-49jc7"]
Jul 08 16:47:51.918594 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.918578    2575 scope.go:117] "RemoveContainer" containerID="8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4"
Jul 08 16:47:51.918851 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:47:51.918822    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4\": container with ID starting with 8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4 not found: ID does not exist" containerID="8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4"
Jul 08 16:47:51.918939 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.918858    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4"} err="failed to get container status \"8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4\": rpc error: code = NotFound desc = could not find container \"8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4\": container with ID starting with 8218b582cdb9929f8fe966e67a597c3db6a53f2f23b4df3084852e4d2b3402c4 not found: ID does not exist"
Jul 08 16:47:51.918939 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.918874    2575 scope.go:117] "RemoveContainer" containerID="8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d"
Jul 08 16:47:51.919078 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:47:51.919059    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d\": container with ID starting with 8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d not found: ID does not exist" containerID="8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d"
Jul 08 16:47:51.919130 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:51.919090    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d"} err="failed to get container status \"8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d\": rpc error: code = NotFound desc = could not find container \"8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d\": container with ID starting with 8d273a7376abfcd9fe5e2ea491bd7a2d0e73351de3a8e3f08aebb45fa689457d not found: ID does not exist"
Jul 08 16:47:53.607841 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:53.607810    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" path="/var/lib/kubelet/pods/356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8/volumes"
Jul 08 16:47:53.608217 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:53.608204    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" path="/var/lib/kubelet/pods/e0161d25-bd08-40da-84ff-a62b4758a1ad/volumes"
Jul 08 16:47:55.877450 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:55.877421    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:47:55.878799 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:55.878759    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 16:47:56.886318 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:56.886289    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:47:56.886813 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:47:56.886783    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:48:05.879024 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:05.878982    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 16:48:06.886860 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:06.886820    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:48:15.878623 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:15.878587    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 16:48:16.887729 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:16.887690    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:48:25.878963 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:25.878918    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 16:48:26.887734 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:26.887689    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:48:35.879798 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:35.879762    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:48:36.888687 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:36.888637    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:48:37.813248 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:37.813215    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:38.156587 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:38.156535    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:47.920408 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:47.920377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:48.044152 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.044121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:48.088413 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088383    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"]
Jul 08 16:48:48.088792 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088777    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container"
Jul 08 16:48:48.088843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088796    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container"
Jul 08 16:48:48.088843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088813    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kube-rbac-proxy"
Jul 08 16:48:48.088843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088819    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kube-rbac-proxy"
Jul 08 16:48:48.088843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088826    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kube-rbac-proxy"
Jul 08 16:48:48.088843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088831    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kube-rbac-proxy"
Jul 08 16:48:48.088843 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088842    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container"
Jul 08 16:48:48.089022 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088848    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container"
Jul 08 16:48:48.089022 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088898    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kube-rbac-proxy"
Jul 08 16:48:48.089022 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088906    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kube-rbac-proxy"
Jul 08 16:48:48.089022 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088913    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="356dfcc5-b1f8-4b29-ac1b-d9525b41c2b8" containerName="kserve-container"
Jul 08 16:48:48.089022 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.088920    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0161d25-bd08-40da-84ff-a62b4758a1ad" containerName="kserve-container"
Jul 08 16:48:48.091078 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.091053    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.094010 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.093984    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fcbfa-kube-rbac-proxy-sar-config\""
Jul 08 16:48:48.094394 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.094368    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fcbfa-predictor-serving-cert\""
Jul 08 16:48:48.102908 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.102869    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"]
Jul 08 16:48:48.125688 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.125661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:48.170471 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.170413    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-proxy-tls\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.170471 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.170456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.170623 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.170486    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g55ck\" (UniqueName: \"kubernetes.io/projected/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-api-access-g55ck\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.201853 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.201829    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"]
Jul 08 16:48:48.204202 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.204188    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.206298 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.206278    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fcbfa-predictor-serving-cert\""
Jul 08 16:48:48.206298 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.206286    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fcbfa-kube-rbac-proxy-sar-config\""
Jul 08 16:48:48.206575 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.206541    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:48.215975 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.215953    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"]
Jul 08 16:48:48.271865 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.271844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g55ck\" (UniqueName: \"kubernetes.io/projected/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-api-access-g55ck\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.271973 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.271900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-proxy-tls\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.271973 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.271932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.272489 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.272463    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.274316 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.274299    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-proxy-tls\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.292064 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.292040    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g55ck\" (UniqueName: \"kubernetes.io/projected/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-api-access-g55ck\") pod \"success-200-isvc-fcbfa-predictor-855b5b6765-49fnz\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.364453 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.364423    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:48.372460 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.372438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660316b3-0555-4e3f-a196-c24c318c42c9-proxy-tls\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.372540 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.372477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjlc\" (UniqueName: \"kubernetes.io/projected/660316b3-0555-4e3f-a196-c24c318c42c9-kube-api-access-xnjlc\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.372583 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.372554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660316b3-0555-4e3f-a196-c24c318c42c9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.400740 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.400710    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:48.464710 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.464610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:48.474318 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.473765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660316b3-0555-4e3f-a196-c24c318c42c9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.474318 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.473814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660316b3-0555-4e3f-a196-c24c318c42c9-proxy-tls\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.474318 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.473856    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjlc\" (UniqueName: \"kubernetes.io/projected/660316b3-0555-4e3f-a196-c24c318c42c9-kube-api-access-xnjlc\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.474917 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.474896    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660316b3-0555-4e3f-a196-c24c318c42c9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.476985 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.476962    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660316b3-0555-4e3f-a196-c24c318c42c9-proxy-tls\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.483330 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.483309    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjlc\" (UniqueName: \"kubernetes.io/projected/660316b3-0555-4e3f-a196-c24c318c42c9-kube-api-access-xnjlc\") pod \"error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.515238 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.515210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:48.526087 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.526034    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"]
Jul 08 16:48:48.528079 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:48:48.528056    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6641dc6f_d7e0_4464_8f9d_a04e461ea2ed.slice/crio-84385a48586094660bc4cb2abd04074125270e1c7118f0e27cc3daeaa124ad48 WatchSource:0}: Error finding container 84385a48586094660bc4cb2abd04074125270e1c7118f0e27cc3daeaa124ad48: Status 404 returned error can't find the container with id 84385a48586094660bc4cb2abd04074125270e1c7118f0e27cc3daeaa124ad48
Jul 08 16:48:48.650673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:48.650638    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"]
Jul 08 16:48:48.652279 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:48:48.652258    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660316b3_0555_4e3f_a196_c24c318c42c9.slice/crio-ce093f13e48f6033a08ee111f034064b64ddb40d0f977f73d6a6995e1471d69e WatchSource:0}: Error finding container ce093f13e48f6033a08ee111f034064b64ddb40d0f977f73d6a6995e1471d69e: Status 404 returned error can't find the container with id ce093f13e48f6033a08ee111f034064b64ddb40d0f977f73d6a6995e1471d69e
Jul 08 16:48:49.121718 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.121677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" event={"ID":"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed","Type":"ContainerStarted","Data":"c9d5cf515ed1a8f243ea867ccdb98a9b118cf4f7efc0f78f14fc2308afb6c350"}
Jul 08 16:48:49.122117 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.121759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" event={"ID":"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed","Type":"ContainerStarted","Data":"145207863fb4777edca659316c57b93f2fe5cea80f05978de746cd9c7f3da822"}
Jul 08 16:48:49.122117 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.121792    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:49.122117 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.121816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" event={"ID":"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed","Type":"ContainerStarted","Data":"84385a48586094660bc4cb2abd04074125270e1c7118f0e27cc3daeaa124ad48"}
Jul 08 16:48:49.123578 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.123554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" event={"ID":"660316b3-0555-4e3f-a196-c24c318c42c9","Type":"ContainerStarted","Data":"3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643"}
Jul 08 16:48:49.123723 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.123598    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" event={"ID":"660316b3-0555-4e3f-a196-c24c318c42c9","Type":"ContainerStarted","Data":"a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c"}
Jul 08 16:48:49.123723 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.123619    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" event={"ID":"660316b3-0555-4e3f-a196-c24c318c42c9","Type":"ContainerStarted","Data":"ce093f13e48f6033a08ee111f034064b64ddb40d0f977f73d6a6995e1471d69e"}
Jul 08 16:48:49.123839 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.123808    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:49.123894 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.123859    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:49.125143 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.125118    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:48:49.142786 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.142746    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podStartSLOduration=1.142730531 podStartE2EDuration="1.142730531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:48:49.141903484 +0000 UTC m=+870.131460277" watchObservedRunningTime="2026-07-08 16:48:49.142730531 +0000 UTC m=+870.132287274"
Jul 08 16:48:49.162659 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:49.162608    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podStartSLOduration=1.162594618 podStartE2EDuration="1.162594618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:48:49.162000195 +0000 UTC m=+870.151556968" watchObservedRunningTime="2026-07-08 16:48:49.162594618 +0000 UTC m=+870.152151362"
Jul 08 16:48:50.128393 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:50.128358    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:48:50.128869 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:50.128497    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:50.129682 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:50.129641    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 08 16:48:51.132751 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:51.132716    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 08 16:48:55.132876 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:55.132844    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:48:55.133348 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:55.133277    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:48:56.137768 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:56.137739    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:48:56.138163 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:48:56.138116    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 08 16:49:05.133301 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:05.133261    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:49:06.138782 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:06.138738    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 08 16:49:15.133690 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:15.133598    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:49:16.139081 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:16.139039    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 08 16:49:19.476819 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:19.476788    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:49:19.480943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:19.480923    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:49:25.133523 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:25.133477    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:49:26.138590 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:26.138555    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 08 16:49:35.134793 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:35.134765    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:49:36.138837 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:36.138802    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:49:48.119715 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:48.119614    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:48.363174 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:49:48.363146    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:25.092417 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:51:25.092384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:36.931817 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:51:36.931784    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:59.295945 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:51:59.295904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:54:19.504950 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:54:19.504922    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:54:19.506869 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:54:19.506847    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:56:12.671694 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:56:12.671638    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:57:02.951404 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:02.951331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:57:03.061965 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.061939    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"]
Jul 08 16:57:03.062172 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.062143    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container" containerID="cri-o://5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35" gracePeriod=30
Jul 08 16:57:03.062308 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.062175    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kube-rbac-proxy" containerID="cri-o://7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a" gracePeriod=30
Jul 08 16:57:03.112796 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.112767    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"]
Jul 08 16:57:03.116323 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.116308    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.118759 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.118742    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-109dc-kube-rbac-proxy-sar-config\""
Jul 08 16:57:03.118902 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.118885    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-109dc-predictor-serving-cert\""
Jul 08 16:57:03.125971 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.125951    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"]
Jul 08 16:57:03.174289 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.174255    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"]
Jul 08 16:57:03.174604 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.174574    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container" containerID="cri-o://cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10" gracePeriod=30
Jul 08 16:57:03.174971 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.174939    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kube-rbac-proxy" containerID="cri-o://59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e" gracePeriod=30
Jul 08 16:57:03.189466 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.189446    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"]
Jul 08 16:57:03.193268 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.193249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.195676 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.195655    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-109dc-predictor-serving-cert\""
Jul 08 16:57:03.195775 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.195681    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-109dc-kube-rbac-proxy-sar-config\""
Jul 08 16:57:03.202064 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.202020    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"]
Jul 08 16:57:03.230553 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.230534    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerID="7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a" exitCode=2
Jul 08 16:57:03.230640 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.230612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" event={"ID":"8ddf3e70-7c90-42fa-9d62-2037f9ec0244","Type":"ContainerDied","Data":"7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a"}
Jul 08 16:57:03.257968 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.257945    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqhr\" (UniqueName: \"kubernetes.io/projected/f0a878fa-162a-4067-b74e-276407d12efa-kube-api-access-hmqhr\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.258046 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.257975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a878fa-162a-4067-b74e-276407d12efa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.258046 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.258015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a878fa-162a-4067-b74e-276407d12efa-proxy-tls\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.358727 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.358706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/415c8967-85e5-46d7-a26a-c92300dbef49-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.358828 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.358746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a878fa-162a-4067-b74e-276407d12efa-proxy-tls\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.358828 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.358798    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68lg6\" (UniqueName: \"kubernetes.io/projected/415c8967-85e5-46d7-a26a-c92300dbef49-kube-api-access-68lg6\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.358903 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.358854    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/415c8967-85e5-46d7-a26a-c92300dbef49-proxy-tls\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.358903 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.358893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqhr\" (UniqueName: \"kubernetes.io/projected/f0a878fa-162a-4067-b74e-276407d12efa-kube-api-access-hmqhr\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.358968 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.358915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a878fa-162a-4067-b74e-276407d12efa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.359454 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.359437    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a878fa-162a-4067-b74e-276407d12efa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.360929 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.360912    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a878fa-162a-4067-b74e-276407d12efa-proxy-tls\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.368098 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.368080    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqhr\" (UniqueName: \"kubernetes.io/projected/f0a878fa-162a-4067-b74e-276407d12efa-kube-api-access-hmqhr\") pod \"success-200-isvc-109dc-predictor-d4554f8b7-4wqp6\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") " pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.426271 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.426249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:03.459992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.459925    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68lg6\" (UniqueName: \"kubernetes.io/projected/415c8967-85e5-46d7-a26a-c92300dbef49-kube-api-access-68lg6\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.459992 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.459968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/415c8967-85e5-46d7-a26a-c92300dbef49-proxy-tls\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.460141 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.460036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/415c8967-85e5-46d7-a26a-c92300dbef49-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.460656 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.460618    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/415c8967-85e5-46d7-a26a-c92300dbef49-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.462251 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.462231    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/415c8967-85e5-46d7-a26a-c92300dbef49-proxy-tls\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.468566 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.468544    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68lg6\" (UniqueName: \"kubernetes.io/projected/415c8967-85e5-46d7-a26a-c92300dbef49-kube-api-access-68lg6\") pod \"error-404-isvc-109dc-predictor-5dbbcdb786-92l9g\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") " pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.504436 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.504404    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:03.559807 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.559696    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"]
Jul 08 16:57:03.641628 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:03.641608    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"]
Jul 08 16:57:03.651739 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:57:03.651715    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415c8967_85e5_46d7_a26a_c92300dbef49.slice/crio-722f3e69adf1ba23b76527ae848a0aa0e9d06426f8a0cd1ef1bc6b734e3a84c7 WatchSource:0}: Error finding container 722f3e69adf1ba23b76527ae848a0aa0e9d06426f8a0cd1ef1bc6b734e3a84c7: Status 404 returned error can't find the container with id 722f3e69adf1ba23b76527ae848a0aa0e9d06426f8a0cd1ef1bc6b734e3a84c7
Jul 08 16:57:04.236458 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.236419    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" event={"ID":"f0a878fa-162a-4067-b74e-276407d12efa","Type":"ContainerStarted","Data":"391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0"}
Jul 08 16:57:04.236946 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.236491    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:04.236946 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.236521    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" event={"ID":"f0a878fa-162a-4067-b74e-276407d12efa","Type":"ContainerStarted","Data":"2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b"}
Jul 08 16:57:04.236946 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.236549    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" event={"ID":"f0a878fa-162a-4067-b74e-276407d12efa","Type":"ContainerStarted","Data":"1373f40e8d19a5e221b956552088be54066a234c27c6ea3ebc545784c05b56a4"}
Jul 08 16:57:04.236946 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.236577    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:04.238245 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.238209    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:04.238425 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.238396    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" event={"ID":"415c8967-85e5-46d7-a26a-c92300dbef49","Type":"ContainerStarted","Data":"c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f"}
Jul 08 16:57:04.238558 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.238459    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:04.238558 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.238492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" event={"ID":"415c8967-85e5-46d7-a26a-c92300dbef49","Type":"ContainerStarted","Data":"c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f"}
Jul 08 16:57:04.238558 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.238522    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" event={"ID":"415c8967-85e5-46d7-a26a-c92300dbef49","Type":"ContainerStarted","Data":"722f3e69adf1ba23b76527ae848a0aa0e9d06426f8a0cd1ef1bc6b734e3a84c7"}
Jul 08 16:57:04.238558 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.238554    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:04.239678 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.239628    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 16:57:04.240181 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.240159    2575 generic.go:356] "Generic (PLEG): container finished" podID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerID="59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e" exitCode=2
Jul 08 16:57:04.240267 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.240209    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" event={"ID":"678b5684-6ab2-4465-9ac7-78f25661f96c","Type":"ContainerDied","Data":"59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e"}
Jul 08 16:57:04.255870 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.255833    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podStartSLOduration=1.255820271 podStartE2EDuration="1.255820271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:57:04.254378839 +0000 UTC m=+1365.243935581" watchObservedRunningTime="2026-07-08 16:57:04.255820271 +0000 UTC m=+1365.245377013"
Jul 08 16:57:04.272524 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.272485    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podStartSLOduration=1.272473879 podStartE2EDuration="1.272473879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:57:04.272076139 +0000 UTC m=+1365.261632915" watchObservedRunningTime="2026-07-08 16:57:04.272473879 +0000 UTC m=+1365.262030687"
Jul 08 16:57:04.346053 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:04.346026    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:57:05.248667 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:05.248617    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:05.249052 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:05.248731    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 16:57:05.872204 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:05.872171    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 08 16:57:05.879696 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:05.879620    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 16:57:06.015034 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.015008    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:57:06.081525 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.081456    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-rbac-proxy-sar-config\") pod \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") "
Jul 08 16:57:06.081525 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.081511    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxp6q\" (UniqueName: \"kubernetes.io/projected/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-api-access-gxp6q\") pod \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") "
Jul 08 16:57:06.081729 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.081546    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls\") pod \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\" (UID: \"8ddf3e70-7c90-42fa-9d62-2037f9ec0244\") "
Jul 08 16:57:06.081885 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.081858    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ddf3e70-7c90-42fa-9d62-2037f9ec0244" (UID: "8ddf3e70-7c90-42fa-9d62-2037f9ec0244"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:57:06.083616 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.083586    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-api-access-gxp6q" (OuterVolumeSpecName: "kube-api-access-gxp6q") pod "8ddf3e70-7c90-42fa-9d62-2037f9ec0244" (UID: "8ddf3e70-7c90-42fa-9d62-2037f9ec0244"). InnerVolumeSpecName "kube-api-access-gxp6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:57:06.083959 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.083929    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ddf3e70-7c90-42fa-9d62-2037f9ec0244" (UID: "8ddf3e70-7c90-42fa-9d62-2037f9ec0244"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:57:06.183062 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.183036    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:57:06.183169 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.183070    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxp6q\" (UniqueName: \"kubernetes.io/projected/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-kube-api-access-gxp6q\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:57:06.183169 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.183088    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ddf3e70-7c90-42fa-9d62-2037f9ec0244-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:57:06.255184 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.255161    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerID="5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35" exitCode=0
Jul 08 16:57:06.255490 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.255210    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"
Jul 08 16:57:06.255490 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.255251    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" event={"ID":"8ddf3e70-7c90-42fa-9d62-2037f9ec0244","Type":"ContainerDied","Data":"5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35"}
Jul 08 16:57:06.255490 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.255319    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk" event={"ID":"8ddf3e70-7c90-42fa-9d62-2037f9ec0244","Type":"ContainerDied","Data":"e674c92e434592e2fc16bd50a76c8adabc42ae49cb12a2104f9b66a972d3f434"}
Jul 08 16:57:06.255490 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.255358    2575 scope.go:117] "RemoveContainer" containerID="7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a"
Jul 08 16:57:06.264139 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.264122    2575 scope.go:117] "RemoveContainer" containerID="5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35"
Jul 08 16:57:06.271803 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.271785    2575 scope.go:117] "RemoveContainer" containerID="7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a"
Jul 08 16:57:06.272045 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:57:06.272025    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a\": container with ID starting with 7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a not found: ID does not exist" containerID="7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a"
Jul 08 16:57:06.272091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.272052    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a"} err="failed to get container status \"7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a\": rpc error: code = NotFound desc = could not find container \"7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a\": container with ID starting with 7932d2444e74aadb650236d6d1ffb91bbd3a9af6c70440830f27e78dd896750a not found: ID does not exist"
Jul 08 16:57:06.272091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.272068    2575 scope.go:117] "RemoveContainer" containerID="5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35"
Jul 08 16:57:06.272294 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:57:06.272279    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35\": container with ID starting with 5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35 not found: ID does not exist" containerID="5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35"
Jul 08 16:57:06.272339 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.272301    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35"} err="failed to get container status \"5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35\": rpc error: code = NotFound desc = could not find container \"5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35\": container with ID starting with 5b63069b7f39c6794236909b6b281f912a56fcfd2118341e5e488d4beb490a35 not found: ID does not exist"
Jul 08 16:57:06.288105 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.288081    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"]
Jul 08 16:57:06.291586 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.291564    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-429af-predictor-7c6dbbb6f7-pjjwk"]
Jul 08 16:57:06.722235 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.722212    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:57:06.787614 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.787590    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/678b5684-6ab2-4465-9ac7-78f25661f96c-proxy-tls\") pod \"678b5684-6ab2-4465-9ac7-78f25661f96c\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") "
Jul 08 16:57:06.787994 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.787628    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msbwm\" (UniqueName: \"kubernetes.io/projected/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-api-access-msbwm\") pod \"678b5684-6ab2-4465-9ac7-78f25661f96c\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") "
Jul 08 16:57:06.787994 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.787677    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-rbac-proxy-sar-config\") pod \"678b5684-6ab2-4465-9ac7-78f25661f96c\" (UID: \"678b5684-6ab2-4465-9ac7-78f25661f96c\") "
Jul 08 16:57:06.788145 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.788009    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "678b5684-6ab2-4465-9ac7-78f25661f96c" (UID: "678b5684-6ab2-4465-9ac7-78f25661f96c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:57:06.789699 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.789669    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/678b5684-6ab2-4465-9ac7-78f25661f96c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "678b5684-6ab2-4465-9ac7-78f25661f96c" (UID: "678b5684-6ab2-4465-9ac7-78f25661f96c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:57:06.789822 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.789792    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-api-access-msbwm" (OuterVolumeSpecName: "kube-api-access-msbwm") pod "678b5684-6ab2-4465-9ac7-78f25661f96c" (UID: "678b5684-6ab2-4465-9ac7-78f25661f96c"). InnerVolumeSpecName "kube-api-access-msbwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:57:06.888454 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.888429    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/678b5684-6ab2-4465-9ac7-78f25661f96c-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:57:06.888454 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.888452    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbwm\" (UniqueName: \"kubernetes.io/projected/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-api-access-msbwm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:57:06.888614 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:06.888462    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/678b5684-6ab2-4465-9ac7-78f25661f96c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:57:07.263302 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.263225    2575 generic.go:356] "Generic (PLEG): container finished" podID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerID="cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10" exitCode=0
Jul 08 16:57:07.263302 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.263283    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"
Jul 08 16:57:07.263808 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.263310    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" event={"ID":"678b5684-6ab2-4465-9ac7-78f25661f96c","Type":"ContainerDied","Data":"cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10"}
Jul 08 16:57:07.263808 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.263410    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs" event={"ID":"678b5684-6ab2-4465-9ac7-78f25661f96c","Type":"ContainerDied","Data":"515f22a120f9d2a9bbea92d656204d24bf314cfc37658d1152bad92723156161"}
Jul 08 16:57:07.263808 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.263452    2575 scope.go:117] "RemoveContainer" containerID="59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e"
Jul 08 16:57:07.271818 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.271798    2575 scope.go:117] "RemoveContainer" containerID="cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10"
Jul 08 16:57:07.280439 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.280425    2575 scope.go:117] "RemoveContainer" containerID="59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e"
Jul 08 16:57:07.280722 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:57:07.280702    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e\": container with ID starting with 59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e not found: ID does not exist" containerID="59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e"
Jul 08 16:57:07.280809 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.280732    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e"} err="failed to get container status \"59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e\": rpc error: code = NotFound desc = could not find container \"59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e\": container with ID starting with 59738b4d46461c8806bc449b4d022e2f7b7db2f72fe0aa0104aed98962ddc64e not found: ID does not exist"
Jul 08 16:57:07.280809 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.280756    2575 scope.go:117] "RemoveContainer" containerID="cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10"
Jul 08 16:57:07.281016 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:57:07.280999    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10\": container with ID starting with cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10 not found: ID does not exist" containerID="cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10"
Jul 08 16:57:07.281069 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.281024    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10"} err="failed to get container status \"cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10\": rpc error: code = NotFound desc = could not find container \"cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10\": container with ID starting with cf8c79766f0e102dc9fab6c5738632bd21374c6bd40342bc99ed20732d13bc10 not found: ID does not exist"
Jul 08 16:57:07.286641 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.286619    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"]
Jul 08 16:57:07.289375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.289356    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-429af-predictor-5679c5df6-n85zs"]
Jul 08 16:57:07.608717 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.608690    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" path="/var/lib/kubelet/pods/678b5684-6ab2-4465-9ac7-78f25661f96c/volumes"
Jul 08 16:57:07.609091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:07.609079    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" path="/var/lib/kubelet/pods/8ddf3e70-7c90-42fa-9d62-2037f9ec0244/volumes"
Jul 08 16:57:10.253541 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:10.253509    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:10.254000 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:10.253903    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:57:10.254088 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:10.254006    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:10.254382 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:10.254357    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 16:57:20.254353 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:20.254309    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:20.254735 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:20.254313    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 16:57:30.253971 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:30.253930    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:30.254374 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:30.254330    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 16:57:40.253897 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:40.253860    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:40.254381 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:40.254341    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 16:57:50.254666 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:50.254616    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:57:50.255554 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:57:50.255536    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:58:02.812850 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.812818    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:02.892736 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.892704    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"]
Jul 08 16:58:02.893007 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.892973    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container" containerID="cri-o://145207863fb4777edca659316c57b93f2fe5cea80f05978de746cd9c7f3da822" gracePeriod=30
Jul 08 16:58:02.893135 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.893007    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kube-rbac-proxy" containerID="cri-o://c9d5cf515ed1a8f243ea867ccdb98a9b118cf4f7efc0f78f14fc2308afb6c350" gracePeriod=30
Jul 08 16:58:02.914920 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.914897    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:02.959391 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.959365    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"]
Jul 08 16:58:02.959709 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.959601    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container" containerID="cri-o://a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c" gracePeriod=30
Jul 08 16:58:02.959709 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.959619    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kube-rbac-proxy" containerID="cri-o://3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643" gracePeriod=30
Jul 08 16:58:02.963518 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963493    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"]
Jul 08 16:58:02.963844 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963829    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container"
Jul 08 16:58:02.963844 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963844    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963853    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kube-rbac-proxy"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963859    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kube-rbac-proxy"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963868    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kube-rbac-proxy"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963873    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kube-rbac-proxy"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963888    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963893    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963954    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kserve-container"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963964    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kserve-container"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963970    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddf3e70-7c90-42fa-9d62-2037f9ec0244" containerName="kube-rbac-proxy"
Jul 08 16:58:02.964003 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.963977    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="678b5684-6ab2-4465-9ac7-78f25661f96c" containerName="kube-rbac-proxy"
Jul 08 16:58:02.967407 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.967391    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:02.969818 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.969800    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-178bc-predictor-serving-cert\""
Jul 08 16:58:02.969921 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.969818    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-178bc-kube-rbac-proxy-sar-config\""
Jul 08 16:58:02.979577 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.979536    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"]
Jul 08 16:58:02.988674 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:02.988639    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:03.068714 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.068689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:03.082135 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.082116    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"]
Jul 08 16:58:03.085388 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.085370    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.089057 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.089038    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-178bc-kube-rbac-proxy-sar-config\""
Jul 08 16:58:03.089143 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.089052    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-178bc-predictor-serving-cert\""
Jul 08 16:58:03.097408 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.097391    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"]
Jul 08 16:58:03.113498 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.113478    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8tf\" (UniqueName: \"kubernetes.io/projected/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-api-access-qh8tf\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.113602 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.113540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.113602 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.113567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.197986 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.197954    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:03.214195 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.214164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-proxy-tls\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.214319 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.214238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.214319 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.214273    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9bkj\" (UniqueName: \"kubernetes.io/projected/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-api-access-z9bkj\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.214319 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.214307    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.214477 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.214362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8tf\" (UniqueName: \"kubernetes.io/projected/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-api-access-qh8tf\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.214477 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.214392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.214477 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:03.214444    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-178bc-predictor-serving-cert: secret "success-200-isvc-178bc-predictor-serving-cert" not found
Jul 08 16:58:03.214581 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:03.214501    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls podName:eafd9458-b52a-488b-9c7a-804b22f35fed nodeName:}" failed. No retries permitted until 2026-07-08 16:58:03.714481808 +0000 UTC m=+1424.704038530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls") pod "success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" (UID: "eafd9458-b52a-488b-9c7a-804b22f35fed") : secret "success-200-isvc-178bc-predictor-serving-cert" not found
Jul 08 16:58:03.214921 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.214903    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.225501 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.225475    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8tf\" (UniqueName: \"kubernetes.io/projected/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-api-access-qh8tf\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.259122 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.259100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:03.315617 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.315587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9bkj\" (UniqueName: \"kubernetes.io/projected/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-api-access-z9bkj\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.315764 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.315675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.315764 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.315713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-proxy-tls\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.316360 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.316341    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.317924 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.317902    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-proxy-tls\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.324884 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.324866    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9bkj\" (UniqueName: \"kubernetes.io/projected/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-api-access-z9bkj\") pod \"error-404-isvc-178bc-predictor-79b66c889d-d8dvk\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") " pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.397066 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.397042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:03.477041 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.477005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:03.519942 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.519909    2575 generic.go:356] "Generic (PLEG): container finished" podID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerID="c9d5cf515ed1a8f243ea867ccdb98a9b118cf4f7efc0f78f14fc2308afb6c350" exitCode=2
Jul 08 16:58:03.520098 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.519964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" event={"ID":"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed","Type":"ContainerDied","Data":"c9d5cf515ed1a8f243ea867ccdb98a9b118cf4f7efc0f78f14fc2308afb6c350"}
Jul 08 16:58:03.521943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.521922    2575 generic.go:356] "Generic (PLEG): container finished" podID="660316b3-0555-4e3f-a196-c24c318c42c9" containerID="3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643" exitCode=2
Jul 08 16:58:03.522054 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.521958    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" event={"ID":"660316b3-0555-4e3f-a196-c24c318c42c9","Type":"ContainerDied","Data":"3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643"}
Jul 08 16:58:03.537683 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.537662    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"]
Jul 08 16:58:03.539542 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:58:03.539517    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd51f3a_9e78_4902_887f_fb6bcda9b5a6.slice/crio-6397b4f800c78492436e7fff2618c242b28546fe9afabc4ba67a6f0eb1a4e596 WatchSource:0}: Error finding container 6397b4f800c78492436e7fff2618c242b28546fe9afabc4ba67a6f0eb1a4e596: Status 404 returned error can't find the container with id 6397b4f800c78492436e7fff2618c242b28546fe9afabc4ba67a6f0eb1a4e596
Jul 08 16:58:03.718531 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.718499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.720635 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.720617    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls\") pod \"success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") " pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.881568 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.881532    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:03.978246 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:03.978195    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:04.015431 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.015408    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"]
Jul 08 16:58:04.017095 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:58:04.017072    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafd9458_b52a_488b_9c7a_804b22f35fed.slice/crio-d73109c719db97ef2e23305b91f7117098a4b18b4c4e2ac8632f4a07ecf71612 WatchSource:0}: Error finding container d73109c719db97ef2e23305b91f7117098a4b18b4c4e2ac8632f4a07ecf71612: Status 404 returned error can't find the container with id d73109c719db97ef2e23305b91f7117098a4b18b4c4e2ac8632f4a07ecf71612
Jul 08 16:58:04.534009 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.533971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" event={"ID":"eafd9458-b52a-488b-9c7a-804b22f35fed","Type":"ContainerStarted","Data":"ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db"}
Jul 08 16:58:04.534228 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.534070    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" event={"ID":"eafd9458-b52a-488b-9c7a-804b22f35fed","Type":"ContainerStarted","Data":"1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15"}
Jul 08 16:58:04.534228 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.534103    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" event={"ID":"eafd9458-b52a-488b-9c7a-804b22f35fed","Type":"ContainerStarted","Data":"d73109c719db97ef2e23305b91f7117098a4b18b4c4e2ac8632f4a07ecf71612"}
Jul 08 16:58:04.534228 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.534133    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:04.534228 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.534163    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:04.537530 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.537491    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 08 16:58:04.538673 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.538623    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" event={"ID":"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6","Type":"ContainerStarted","Data":"452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3"}
Jul 08 16:58:04.538862 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.538695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" event={"ID":"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6","Type":"ContainerStarted","Data":"b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1"}
Jul 08 16:58:04.538862 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.538724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" event={"ID":"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6","Type":"ContainerStarted","Data":"6397b4f800c78492436e7fff2618c242b28546fe9afabc4ba67a6f0eb1a4e596"}
Jul 08 16:58:04.538862 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.538755    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:04.538862 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.538781    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:04.540018 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.539989    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 08 16:58:04.555589 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.555548    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podStartSLOduration=2.555534721 podStartE2EDuration="2.555534721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:58:04.553552784 +0000 UTC m=+1425.543109527" watchObservedRunningTime="2026-07-08 16:58:04.555534721 +0000 UTC m=+1425.545091509"
Jul 08 16:58:04.574858 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:04.574821    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podStartSLOduration=1.574807482 podStartE2EDuration="1.574807482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:58:04.573982831 +0000 UTC m=+1425.563539584" watchObservedRunningTime="2026-07-08 16:58:04.574807482 +0000 UTC m=+1425.564364225"
Jul 08 16:58:05.129036 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:05.128997    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 08 16:58:05.133334 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:05.133309    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:58:05.544332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:05.544241    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 08 16:58:05.544477 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:05.544344    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 08 16:58:06.133725 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.133683    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 08 16:58:06.139097 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.139066    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 08 16:58:06.549781 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.549751    2575 generic.go:356] "Generic (PLEG): container finished" podID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerID="145207863fb4777edca659316c57b93f2fe5cea80f05978de746cd9c7f3da822" exitCode=0
Jul 08 16:58:06.549883 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.549820    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" event={"ID":"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed","Type":"ContainerDied","Data":"145207863fb4777edca659316c57b93f2fe5cea80f05978de746cd9c7f3da822"}
Jul 08 16:58:06.582437 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.582409    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:58:06.701435 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.701412    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:58:06.746195 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.746172    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-proxy-tls\") pod \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") "
Jul 08 16:58:06.746354 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.746280    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g55ck\" (UniqueName: \"kubernetes.io/projected/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-api-access-g55ck\") pod \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") "
Jul 08 16:58:06.746354 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.746313    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-rbac-proxy-sar-config\") pod \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\" (UID: \"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed\") "
Jul 08 16:58:06.746793 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.746757    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" (UID: "6641dc6f-d7e0-4464-8f9d-a04e461ea2ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:58:06.748706 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.748687    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" (UID: "6641dc6f-d7e0-4464-8f9d-a04e461ea2ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:58:06.748826 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.748803    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-api-access-g55ck" (OuterVolumeSpecName: "kube-api-access-g55ck") pod "6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" (UID: "6641dc6f-d7e0-4464-8f9d-a04e461ea2ed"). InnerVolumeSpecName "kube-api-access-g55ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:58:06.847068 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.847033    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660316b3-0555-4e3f-a196-c24c318c42c9-kube-rbac-proxy-sar-config\") pod \"660316b3-0555-4e3f-a196-c24c318c42c9\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") "
Jul 08 16:58:06.847068 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.847075    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnjlc\" (UniqueName: \"kubernetes.io/projected/660316b3-0555-4e3f-a196-c24c318c42c9-kube-api-access-xnjlc\") pod \"660316b3-0555-4e3f-a196-c24c318c42c9\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") "
Jul 08 16:58:06.847236 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.847140    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660316b3-0555-4e3f-a196-c24c318c42c9-proxy-tls\") pod \"660316b3-0555-4e3f-a196-c24c318c42c9\" (UID: \"660316b3-0555-4e3f-a196-c24c318c42c9\") "
Jul 08 16:58:06.847300 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.847288    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:06.847354 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.847301    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g55ck\" (UniqueName: \"kubernetes.io/projected/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-api-access-g55ck\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:06.847354 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.847310    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:06.847354 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.847318    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660316b3-0555-4e3f-a196-c24c318c42c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "660316b3-0555-4e3f-a196-c24c318c42c9" (UID: "660316b3-0555-4e3f-a196-c24c318c42c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:58:06.849020 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.848998    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660316b3-0555-4e3f-a196-c24c318c42c9-kube-api-access-xnjlc" (OuterVolumeSpecName: "kube-api-access-xnjlc") pod "660316b3-0555-4e3f-a196-c24c318c42c9" (UID: "660316b3-0555-4e3f-a196-c24c318c42c9"). InnerVolumeSpecName "kube-api-access-xnjlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:58:06.849106 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.849048    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660316b3-0555-4e3f-a196-c24c318c42c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "660316b3-0555-4e3f-a196-c24c318c42c9" (UID: "660316b3-0555-4e3f-a196-c24c318c42c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:58:06.947838 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.947784    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660316b3-0555-4e3f-a196-c24c318c42c9-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:06.947838 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.947804    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660316b3-0555-4e3f-a196-c24c318c42c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:06.947838 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:06.947814    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnjlc\" (UniqueName: \"kubernetes.io/projected/660316b3-0555-4e3f-a196-c24c318c42c9-kube-api-access-xnjlc\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:07.556719 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.556692    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"
Jul 08 16:58:07.556719 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.556710    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz" event={"ID":"6641dc6f-d7e0-4464-8f9d-a04e461ea2ed","Type":"ContainerDied","Data":"84385a48586094660bc4cb2abd04074125270e1c7118f0e27cc3daeaa124ad48"}
Jul 08 16:58:07.557210 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.556776    2575 scope.go:117] "RemoveContainer" containerID="c9d5cf515ed1a8f243ea867ccdb98a9b118cf4f7efc0f78f14fc2308afb6c350"
Jul 08 16:58:07.558301 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.558274    2575 generic.go:356] "Generic (PLEG): container finished" podID="660316b3-0555-4e3f-a196-c24c318c42c9" containerID="a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c" exitCode=0
Jul 08 16:58:07.558446 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.558322    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"
Jul 08 16:58:07.558446 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.558349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" event={"ID":"660316b3-0555-4e3f-a196-c24c318c42c9","Type":"ContainerDied","Data":"a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c"}
Jul 08 16:58:07.558446 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.558395    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b" event={"ID":"660316b3-0555-4e3f-a196-c24c318c42c9","Type":"ContainerDied","Data":"ce093f13e48f6033a08ee111f034064b64ddb40d0f977f73d6a6995e1471d69e"}
Jul 08 16:58:07.566393 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.566343    2575 scope.go:117] "RemoveContainer" containerID="145207863fb4777edca659316c57b93f2fe5cea80f05978de746cd9c7f3da822"
Jul 08 16:58:07.574176 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.574161    2575 scope.go:117] "RemoveContainer" containerID="3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643"
Jul 08 16:58:07.582965 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.582940    2575 scope.go:117] "RemoveContainer" containerID="a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c"
Jul 08 16:58:07.585416 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.585394    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"]
Jul 08 16:58:07.590523 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.590503    2575 scope.go:117] "RemoveContainer" containerID="3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643"
Jul 08 16:58:07.591066 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:07.590959    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643\": container with ID starting with 3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643 not found: ID does not exist" containerID="3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643"
Jul 08 16:58:07.591066 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.590993    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643"} err="failed to get container status \"3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643\": rpc error: code = NotFound desc = could not find container \"3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643\": container with ID starting with 3d51a6749f6b5f4d9a200e8a59e4186224777cf34c7c8788df3f012fa2348643 not found: ID does not exist"
Jul 08 16:58:07.591066 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.591015    2575 scope.go:117] "RemoveContainer" containerID="a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c"
Jul 08 16:58:07.591287 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:07.591269    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c\": container with ID starting with a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c not found: ID does not exist" containerID="a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c"
Jul 08 16:58:07.591336 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.591291    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c"} err="failed to get container status \"a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c\": rpc error: code = NotFound desc = could not find container \"a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c\": container with ID starting with a05dd8df8fd12de396d865cb862aa89fdd6b161d589f1ea8232b3e196183c00c not found: ID does not exist"
Jul 08 16:58:07.592445 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.592428    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbfa-predictor-855b5b6765-49fnz"]
Jul 08 16:58:07.609038 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.609017    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" path="/var/lib/kubelet/pods/6641dc6f-d7e0-4464-8f9d-a04e461ea2ed/volumes"
Jul 08 16:58:07.609301 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.609289    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"]
Jul 08 16:58:07.609696 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:07.609676    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbfa-predictor-694c77d57b-zxw5b"]
Jul 08 16:58:09.614136 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:09.614104    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" path="/var/lib/kubelet/pods/660316b3-0555-4e3f-a196-c24c318c42c9/volumes"
Jul 08 16:58:10.548252 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:10.548220    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:10.548751 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:10.548726    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:10.548876 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:10.548816    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 08 16:58:10.549139 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:10.549116    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 08 16:58:13.288035 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.287959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:13.353028 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.352998    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"]
Jul 08 16:58:13.353233 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.353207    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container" containerID="cri-o://c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f" gracePeriod=30
Jul 08 16:58:13.353362 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.353242    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kube-rbac-proxy" containerID="cri-o://c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f" gracePeriod=30
Jul 08 16:58:13.376801 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.376776    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:13.418495 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.418468    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"]
Jul 08 16:58:13.418773 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.418743    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container" containerID="cri-o://2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b" gracePeriod=30
Jul 08 16:58:13.418907 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.418784    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kube-rbac-proxy" containerID="cri-o://391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0" gracePeriod=30
Jul 08 16:58:13.447393 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447366    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"]
Jul 08 16:58:13.447731 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447719    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kube-rbac-proxy"
Jul 08 16:58:13.447784 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447732    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kube-rbac-proxy"
Jul 08 16:58:13.447784 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447754    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kube-rbac-proxy"
Jul 08 16:58:13.447784 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447760    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kube-rbac-proxy"
Jul 08 16:58:13.447784 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447771    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container"
Jul 08 16:58:13.447784 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447776    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container"
Jul 08 16:58:13.447784 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447785    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container"
Jul 08 16:58:13.447964 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447791    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container"
Jul 08 16:58:13.447964 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447843    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kserve-container"
Jul 08 16:58:13.447964 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447850    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kserve-container"
Jul 08 16:58:13.447964 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447857    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="660316b3-0555-4e3f-a196-c24c318c42c9" containerName="kube-rbac-proxy"
Jul 08 16:58:13.447964 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.447865    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6641dc6f-d7e0-4464-8f9d-a04e461ea2ed" containerName="kube-rbac-proxy"
Jul 08 16:58:13.452518 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.452501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.454021 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.454001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:13.454868 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.454853    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-806a8-predictor-serving-cert\""
Jul 08 16:58:13.454941 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.454868    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-806a8-kube-rbac-proxy-sar-config\""
Jul 08 16:58:13.460417 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.460398    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"]
Jul 08 16:58:13.499042 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.499012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b45dea-ae5b-4b61-9b05-209978bc4a1f-proxy-tls\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.499178 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.499152    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.499241 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.499231    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89zv\" (UniqueName: \"kubernetes.io/projected/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-api-access-w89zv\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.523347 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.523320    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:13.541044 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.540992    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"]
Jul 08 16:58:13.544329 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.544313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.546615 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.546593    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-806a8-kube-rbac-proxy-sar-config\""
Jul 08 16:58:13.546735 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.546695    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-806a8-predictor-serving-cert\""
Jul 08 16:58:13.555080 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.555059    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"]
Jul 08 16:58:13.591979 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.591951    2575 generic.go:356] "Generic (PLEG): container finished" podID="415c8967-85e5-46d7-a26a-c92300dbef49" containerID="c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f" exitCode=2
Jul 08 16:58:13.592095 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.592029    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" event={"ID":"415c8967-85e5-46d7-a26a-c92300dbef49","Type":"ContainerDied","Data":"c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f"}
Jul 08 16:58:13.593678 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.593637    2575 generic.go:356] "Generic (PLEG): container finished" podID="f0a878fa-162a-4067-b74e-276407d12efa" containerID="391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0" exitCode=2
Jul 08 16:58:13.593769 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.593703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" event={"ID":"f0a878fa-162a-4067-b74e-276407d12efa","Type":"ContainerDied","Data":"391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0"}
Jul 08 16:58:13.600189 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.600170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxvj\" (UniqueName: \"kubernetes.io/projected/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-api-access-hzxvj\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.600255 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.600210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89zv\" (UniqueName: \"kubernetes.io/projected/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-api-access-w89zv\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.600255 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.600236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b45dea-ae5b-4b61-9b05-209978bc4a1f-proxy-tls\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.600322 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.600274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.600322 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.600310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.600397 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.600330    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.601037 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.601019    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.602794 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.602775    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b45dea-ae5b-4b61-9b05-209978bc4a1f-proxy-tls\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.608090 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.608067    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89zv\" (UniqueName: \"kubernetes.io/projected/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-api-access-w89zv\") pod \"success-200-isvc-806a8-predictor-d7dcd8679-j829p\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") " pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.624237 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.624216    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:13.700661 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.700614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.700760 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.700666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxvj\" (UniqueName: \"kubernetes.io/projected/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-api-access-hzxvj\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.700760 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.700713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.700924 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:13.700906    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-806a8-predictor-serving-cert: secret "error-404-isvc-806a8-predictor-serving-cert" not found
Jul 08 16:58:13.700986 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:13.700974    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls podName:d176335b-e0f2-4943-8d7d-4bd699d52418 nodeName:}" failed. No retries permitted until 2026-07-08 16:58:14.200956258 +0000 UTC m=+1435.190512986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls") pod "error-404-isvc-806a8-predictor-5595d45646-t22t9" (UID: "d176335b-e0f2-4943-8d7d-4bd699d52418") : secret "error-404-isvc-806a8-predictor-serving-cert" not found
Jul 08 16:58:13.701207 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.701190    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.709288 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.709269    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxvj\" (UniqueName: \"kubernetes.io/projected/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-api-access-hzxvj\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:13.763509 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.763490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:13.893097 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:13.893071    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"]
Jul 08 16:58:13.895886 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:58:13.895863    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b45dea_ae5b_4b61_9b05_209978bc4a1f.slice/crio-f1454533baccddbf9ee546243ae67954c5b21067952b863f1c57668164a74a55 WatchSource:0}: Error finding container f1454533baccddbf9ee546243ae67954c5b21067952b863f1c57668164a74a55: Status 404 returned error can't find the container with id f1454533baccddbf9ee546243ae67954c5b21067952b863f1c57668164a74a55
Jul 08 16:58:14.173097 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.173065    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:14.204276 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.204250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:14.206452 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.206435    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls\") pod \"error-404-isvc-806a8-predictor-5595d45646-t22t9\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") " pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:14.454906 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.454827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:14.586976 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.586950    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"]
Jul 08 16:58:14.589242 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:58:14.589211    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd176335b_e0f2_4943_8d7d_4bd699d52418.slice/crio-3542b7e6cd9d3303061e34af7f48044901a4af0a763aea4407c6812510bfda0d WatchSource:0}: Error finding container 3542b7e6cd9d3303061e34af7f48044901a4af0a763aea4407c6812510bfda0d: Status 404 returned error can't find the container with id 3542b7e6cd9d3303061e34af7f48044901a4af0a763aea4407c6812510bfda0d
Jul 08 16:58:14.599538 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.599509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" event={"ID":"d176335b-e0f2-4943-8d7d-4bd699d52418","Type":"ContainerStarted","Data":"3542b7e6cd9d3303061e34af7f48044901a4af0a763aea4407c6812510bfda0d"}
Jul 08 16:58:14.601240 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.601215    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" event={"ID":"16b45dea-ae5b-4b61-9b05-209978bc4a1f","Type":"ContainerStarted","Data":"f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5"}
Jul 08 16:58:14.601309 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.601268    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" event={"ID":"16b45dea-ae5b-4b61-9b05-209978bc4a1f","Type":"ContainerStarted","Data":"44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99"}
Jul 08 16:58:14.601309 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.601299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" event={"ID":"16b45dea-ae5b-4b61-9b05-209978bc4a1f","Type":"ContainerStarted","Data":"f1454533baccddbf9ee546243ae67954c5b21067952b863f1c57668164a74a55"}
Jul 08 16:58:14.601467 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.601447    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:14.601523 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.601513    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:14.602919 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.602894    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 08 16:58:14.620431 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:14.620393    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podStartSLOduration=1.620380428 podStartE2EDuration="1.620380428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:58:14.619350922 +0000 UTC m=+1435.608907693" watchObservedRunningTime="2026-07-08 16:58:14.620380428 +0000 UTC m=+1435.609937169"
Jul 08 16:58:15.249376 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:15.249333    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 08 16:58:15.249570 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:15.249412    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 08 16:58:15.608792 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:15.608745    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 08 16:58:15.609333 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:15.609239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" event={"ID":"d176335b-e0f2-4943-8d7d-4bd699d52418","Type":"ContainerStarted","Data":"14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b"}
Jul 08 16:58:15.609333 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:15.609287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" event={"ID":"d176335b-e0f2-4943-8d7d-4bd699d52418","Type":"ContainerStarted","Data":"82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219"}
Jul 08 16:58:15.627817 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:15.627774    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podStartSLOduration=2.627759509 podStartE2EDuration="2.627759509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:58:15.626799619 +0000 UTC m=+1436.616356382" watchObservedRunningTime="2026-07-08 16:58:15.627759509 +0000 UTC m=+1436.617316246"
Jul 08 16:58:16.410829 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.410805    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:58:16.418456 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.418438    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/415c8967-85e5-46d7-a26a-c92300dbef49-proxy-tls\") pod \"415c8967-85e5-46d7-a26a-c92300dbef49\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") "
Jul 08 16:58:16.418533 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.418492    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/415c8967-85e5-46d7-a26a-c92300dbef49-kube-rbac-proxy-sar-config\") pod \"415c8967-85e5-46d7-a26a-c92300dbef49\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") "
Jul 08 16:58:16.418533 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.418529    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68lg6\" (UniqueName: \"kubernetes.io/projected/415c8967-85e5-46d7-a26a-c92300dbef49-kube-api-access-68lg6\") pod \"415c8967-85e5-46d7-a26a-c92300dbef49\" (UID: \"415c8967-85e5-46d7-a26a-c92300dbef49\") "
Jul 08 16:58:16.418923 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.418892    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415c8967-85e5-46d7-a26a-c92300dbef49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "415c8967-85e5-46d7-a26a-c92300dbef49" (UID: "415c8967-85e5-46d7-a26a-c92300dbef49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:58:16.420443 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.420414    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415c8967-85e5-46d7-a26a-c92300dbef49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "415c8967-85e5-46d7-a26a-c92300dbef49" (UID: "415c8967-85e5-46d7-a26a-c92300dbef49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:58:16.420520 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.420414    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415c8967-85e5-46d7-a26a-c92300dbef49-kube-api-access-68lg6" (OuterVolumeSpecName: "kube-api-access-68lg6") pod "415c8967-85e5-46d7-a26a-c92300dbef49" (UID: "415c8967-85e5-46d7-a26a-c92300dbef49"). InnerVolumeSpecName "kube-api-access-68lg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:58:16.519077 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.519002    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/415c8967-85e5-46d7-a26a-c92300dbef49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:16.519077 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.519030    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68lg6\" (UniqueName: \"kubernetes.io/projected/415c8967-85e5-46d7-a26a-c92300dbef49-kube-api-access-68lg6\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:16.519077 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.519040    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/415c8967-85e5-46d7-a26a-c92300dbef49-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:16.614076 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.614046    2575 generic.go:356] "Generic (PLEG): container finished" podID="415c8967-85e5-46d7-a26a-c92300dbef49" containerID="c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f" exitCode=0
Jul 08 16:58:16.614422 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.614097    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"
Jul 08 16:58:16.614422 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.614129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" event={"ID":"415c8967-85e5-46d7-a26a-c92300dbef49","Type":"ContainerDied","Data":"c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f"}
Jul 08 16:58:16.614422 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.614190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g" event={"ID":"415c8967-85e5-46d7-a26a-c92300dbef49","Type":"ContainerDied","Data":"722f3e69adf1ba23b76527ae848a0aa0e9d06426f8a0cd1ef1bc6b734e3a84c7"}
Jul 08 16:58:16.614422 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.614235    2575 scope.go:117] "RemoveContainer" containerID="c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f"
Jul 08 16:58:16.614625 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.614611    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:16.614700 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.614679    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:16.616354 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.616326    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 08 16:58:16.623385 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.623176    2575 scope.go:117] "RemoveContainer" containerID="c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f"
Jul 08 16:58:16.631218 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.631146    2575 scope.go:117] "RemoveContainer" containerID="c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f"
Jul 08 16:58:16.631696 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:16.631555    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f\": container with ID starting with c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f not found: ID does not exist" containerID="c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f"
Jul 08 16:58:16.631696 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.631592    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f"} err="failed to get container status \"c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f\": rpc error: code = NotFound desc = could not find container \"c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f\": container with ID starting with c5e51b0e0ad039a3b3a7fdf1de47daf13d6f86c97c06f179b0095948ac1a9d9f not found: ID does not exist"
Jul 08 16:58:16.631696 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.631612    2575 scope.go:117] "RemoveContainer" containerID="c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f"
Jul 08 16:58:16.632053 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:16.631998    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f\": container with ID starting with c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f not found: ID does not exist" containerID="c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f"
Jul 08 16:58:16.632053 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.632025    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f"} err="failed to get container status \"c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f\": rpc error: code = NotFound desc = could not find container \"c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f\": container with ID starting with c9b41bb687e2dc16c8d180c0bb4b1bab47169089fb15c7bd833c95750563068f not found: ID does not exist"
Jul 08 16:58:16.637504 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.637485    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"]
Jul 08 16:58:16.641432 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:16.641412    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-109dc-predictor-5dbbcdb786-92l9g"]
Jul 08 16:58:17.065660 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.065628    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:58:17.123750 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.123721    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a878fa-162a-4067-b74e-276407d12efa-kube-rbac-proxy-sar-config\") pod \"f0a878fa-162a-4067-b74e-276407d12efa\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") "
Jul 08 16:58:17.123894 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.123772    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a878fa-162a-4067-b74e-276407d12efa-proxy-tls\") pod \"f0a878fa-162a-4067-b74e-276407d12efa\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") "
Jul 08 16:58:17.123894 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.123879    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmqhr\" (UniqueName: \"kubernetes.io/projected/f0a878fa-162a-4067-b74e-276407d12efa-kube-api-access-hmqhr\") pod \"f0a878fa-162a-4067-b74e-276407d12efa\" (UID: \"f0a878fa-162a-4067-b74e-276407d12efa\") "
Jul 08 16:58:17.124106 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.124083    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a878fa-162a-4067-b74e-276407d12efa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0a878fa-162a-4067-b74e-276407d12efa" (UID: "f0a878fa-162a-4067-b74e-276407d12efa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:58:17.125718 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.125696    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a878fa-162a-4067-b74e-276407d12efa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0a878fa-162a-4067-b74e-276407d12efa" (UID: "f0a878fa-162a-4067-b74e-276407d12efa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:58:17.125801 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.125728    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a878fa-162a-4067-b74e-276407d12efa-kube-api-access-hmqhr" (OuterVolumeSpecName: "kube-api-access-hmqhr") pod "f0a878fa-162a-4067-b74e-276407d12efa" (UID: "f0a878fa-162a-4067-b74e-276407d12efa"). InnerVolumeSpecName "kube-api-access-hmqhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:58:17.225310 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.225228    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a878fa-162a-4067-b74e-276407d12efa-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:17.225310 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.225266    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmqhr\" (UniqueName: \"kubernetes.io/projected/f0a878fa-162a-4067-b74e-276407d12efa-kube-api-access-hmqhr\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:17.225310 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.225280    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a878fa-162a-4067-b74e-276407d12efa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:58:17.609717 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.609686    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" path="/var/lib/kubelet/pods/415c8967-85e5-46d7-a26a-c92300dbef49/volumes"
Jul 08 16:58:17.621504 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.621482    2575 generic.go:356] "Generic (PLEG): container finished" podID="f0a878fa-162a-4067-b74e-276407d12efa" containerID="2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b" exitCode=0
Jul 08 16:58:17.621866 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.621528    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"
Jul 08 16:58:17.621866 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.621557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" event={"ID":"f0a878fa-162a-4067-b74e-276407d12efa","Type":"ContainerDied","Data":"2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b"}
Jul 08 16:58:17.621866 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.621609    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6" event={"ID":"f0a878fa-162a-4067-b74e-276407d12efa","Type":"ContainerDied","Data":"1373f40e8d19a5e221b956552088be54066a234c27c6ea3ebc545784c05b56a4"}
Jul 08 16:58:17.621866 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.621641    2575 scope.go:117] "RemoveContainer" containerID="391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0"
Jul 08 16:58:17.622993 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.622965    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 08 16:58:17.630217 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.630191    2575 scope.go:117] "RemoveContainer" containerID="2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b"
Jul 08 16:58:17.637352 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.637333    2575 scope.go:117] "RemoveContainer" containerID="391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0"
Jul 08 16:58:17.637612 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:17.637593    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0\": container with ID starting with 391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0 not found: ID does not exist" containerID="391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0"
Jul 08 16:58:17.637699 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.637619    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0"} err="failed to get container status \"391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0\": rpc error: code = NotFound desc = could not find container \"391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0\": container with ID starting with 391ff1a458f4f589226c95a4bfdd7960d6fcc5919371efca4124e3b92c847ba0 not found: ID does not exist"
Jul 08 16:58:17.637699 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.637636    2575 scope.go:117] "RemoveContainer" containerID="2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b"
Jul 08 16:58:17.637897 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:58:17.637883    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b\": container with ID starting with 2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b not found: ID does not exist" containerID="2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b"
Jul 08 16:58:17.637938 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.637900    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b"} err="failed to get container status \"2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b\": rpc error: code = NotFound desc = could not find container \"2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b\": container with ID starting with 2a2b1465073fbfee99039eecb7d5a7e8405db6c113a77370da323bd80a5d501b not found: ID does not exist"
Jul 08 16:58:17.641419 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.641396    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"]
Jul 08 16:58:17.645631 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:17.645613    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-109dc-predictor-d4554f8b7-4wqp6"]
Jul 08 16:58:19.609318 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:19.609285    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0a878fa-162a-4067-b74e-276407d12efa" path="/var/lib/kubelet/pods/f0a878fa-162a-4067-b74e-276407d12efa/volumes"
Jul 08 16:58:20.548936 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:20.548898    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 08 16:58:20.549162 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:20.549135    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 08 16:58:20.614462 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:20.614424    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:58:20.615095 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:20.615068    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 08 16:58:22.627152 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:22.627125    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:58:22.627630 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:22.627603    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 08 16:58:30.549599 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:30.549559    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 08 16:58:30.550082 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:30.549559    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 08 16:58:30.615720 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:30.615685    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 08 16:58:32.627815 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:32.627774    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 08 16:58:40.548830 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:40.548778    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 08 16:58:40.549341 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:40.549124    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 08 16:58:40.615836 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:40.615803    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 08 16:58:42.627795 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:42.627756    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 08 16:58:50.549743 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:50.549711    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:58:50.550285 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:50.550154    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:58:50.615321 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:50.615292    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 08 16:58:52.627638 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:58:52.627594    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 08 16:59:00.616421 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:00.616394    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 16:59:02.628486 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:02.628457    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 16:59:03.006420 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:03.006334    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:03.126822 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:03.126789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:13.091847 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.091805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:13.160855 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.160826    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"]
Jul 08 16:59:13.161088 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.161047    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container" containerID="cri-o://b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1" gracePeriod=30
Jul 08 16:59:13.161237 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.161110    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kube-rbac-proxy" containerID="cri-o://452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3" gracePeriod=30
Jul 08 16:59:13.184752 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.184731    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:13.204852 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.204829    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"]
Jul 08 16:59:13.205087 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.205064    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container" containerID="cri-o://1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15" gracePeriod=30
Jul 08 16:59:13.205191 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.205106    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kube-rbac-proxy" containerID="cri-o://ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db" gracePeriod=30
Jul 08 16:59:13.252889 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.252866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:13.260455 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260433    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"]
Jul 08 16:59:13.260829 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260814    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kube-rbac-proxy"
Jul 08 16:59:13.260891 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260831    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kube-rbac-proxy"
Jul 08 16:59:13.260891 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260845    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container"
Jul 08 16:59:13.260891 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260850    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container"
Jul 08 16:59:13.260891 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260866    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kube-rbac-proxy"
Jul 08 16:59:13.260891 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260872    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kube-rbac-proxy"
Jul 08 16:59:13.260891 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260880    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container"
Jul 08 16:59:13.260891 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260885    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container"
Jul 08 16:59:13.261103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260950    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kserve-container"
Jul 08 16:59:13.261103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260957    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="415c8967-85e5-46d7-a26a-c92300dbef49" containerName="kube-rbac-proxy"
Jul 08 16:59:13.261103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260965    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kserve-container"
Jul 08 16:59:13.261103 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.260969    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a878fa-162a-4067-b74e-276407d12efa" containerName="kube-rbac-proxy"
Jul 08 16:59:13.265298 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.265279    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.267703 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.267685    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b2f8c-predictor-serving-cert\""
Jul 08 16:59:13.267797 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.267685    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b2f8c-kube-rbac-proxy-sar-config\""
Jul 08 16:59:13.275429 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.275407    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"]
Jul 08 16:59:13.346576 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.346518    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"]
Jul 08 16:59:13.349778 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.349762    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.351958 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.351937    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b2f8c-predictor-serving-cert\""
Jul 08 16:59:13.352055 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.352023    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b2f8c-kube-rbac-proxy-sar-config\""
Jul 08 16:59:13.355602 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.355579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.355725 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.355633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nl7r\" (UniqueName: \"kubernetes.io/projected/e717f006-ba23-401e-9efe-422da3ea256e-kube-api-access-8nl7r\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.355790 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.355730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e717f006-ba23-401e-9efe-422da3ea256e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.359067 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.359047    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"]
Jul 08 16:59:13.366058 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.366040    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:13.456237 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.456207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099312d7-f64f-48a3-801e-18d3e9055d25-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.456339 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.456243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nl7r\" (UniqueName: \"kubernetes.io/projected/e717f006-ba23-401e-9efe-422da3ea256e-kube-api-access-8nl7r\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.456339 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.456276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv578\" (UniqueName: \"kubernetes.io/projected/099312d7-f64f-48a3-801e-18d3e9055d25-kube-api-access-hv578\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.456416 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.456390    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099312d7-f64f-48a3-801e-18d3e9055d25-proxy-tls\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.456449 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.456437    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e717f006-ba23-401e-9efe-422da3ea256e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.456502 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.456491    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.456578 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:59:13.456560    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-serving-cert: secret "success-200-isvc-b2f8c-predictor-serving-cert" not found
Jul 08 16:59:13.456631 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:59:13.456621    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls podName:e717f006-ba23-401e-9efe-422da3ea256e nodeName:}" failed. No retries permitted until 2026-07-08 16:59:13.956602129 +0000 UTC m=+1494.946158863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls") pod "success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" (UID: "e717f006-ba23-401e-9efe-422da3ea256e") : secret "success-200-isvc-b2f8c-predictor-serving-cert" not found
Jul 08 16:59:13.457108 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.457090    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e717f006-ba23-401e-9efe-422da3ea256e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.464599 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.464578    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nl7r\" (UniqueName: \"kubernetes.io/projected/e717f006-ba23-401e-9efe-422da3ea256e-kube-api-access-8nl7r\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.513065 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.513039    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:13.557387 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.557358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv578\" (UniqueName: \"kubernetes.io/projected/099312d7-f64f-48a3-801e-18d3e9055d25-kube-api-access-hv578\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.557504 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.557397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099312d7-f64f-48a3-801e-18d3e9055d25-proxy-tls\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.557572 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.557531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099312d7-f64f-48a3-801e-18d3e9055d25-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.558163 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.558140    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099312d7-f64f-48a3-801e-18d3e9055d25-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.559880 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.559857    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099312d7-f64f-48a3-801e-18d3e9055d25-proxy-tls\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.569619 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.569599    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv578\" (UniqueName: \"kubernetes.io/projected/099312d7-f64f-48a3-801e-18d3e9055d25-kube-api-access-hv578\") pod \"error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") " pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.591959 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.591923    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:13.659840 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.659817    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:13.688492 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.688471    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:13.786061 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.786027    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"]
Jul 08 16:59:13.789597 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:59:13.789574    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099312d7_f64f_48a3_801e_18d3e9055d25.slice/crio-bc141e1f2b6d3ee50b2b13ae43d7f2567accec4dcff1031c967783e7d294570a WatchSource:0}: Error finding container bc141e1f2b6d3ee50b2b13ae43d7f2567accec4dcff1031c967783e7d294570a: Status 404 returned error can't find the container with id bc141e1f2b6d3ee50b2b13ae43d7f2567accec4dcff1031c967783e7d294570a
Jul 08 16:59:13.886323 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.886298    2575 generic.go:356] "Generic (PLEG): container finished" podID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerID="ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db" exitCode=2
Jul 08 16:59:13.886417 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.886386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" event={"ID":"eafd9458-b52a-488b-9c7a-804b22f35fed","Type":"ContainerDied","Data":"ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db"}
Jul 08 16:59:13.887694 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.887672    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" event={"ID":"099312d7-f64f-48a3-801e-18d3e9055d25","Type":"ContainerStarted","Data":"e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb"}
Jul 08 16:59:13.887787 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.887728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" event={"ID":"099312d7-f64f-48a3-801e-18d3e9055d25","Type":"ContainerStarted","Data":"bc141e1f2b6d3ee50b2b13ae43d7f2567accec4dcff1031c967783e7d294570a"}
Jul 08 16:59:13.889431 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.889409    2575 generic.go:356] "Generic (PLEG): container finished" podID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerID="452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3" exitCode=2
Jul 08 16:59:13.889551 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.889456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" event={"ID":"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6","Type":"ContainerDied","Data":"452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3"}
Jul 08 16:59:13.961332 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.961278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:13.963421 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:13.963400    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls\") pod \"success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") " pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:14.176842 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.176776    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:14.303308 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.303282    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"]
Jul 08 16:59:14.305253 ip-10-0-135-143 kubenswrapper[2575]: W0708 16:59:14.305229    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode717f006_ba23_401e_9efe_422da3ea256e.slice/crio-4f1676d38524b8e20b003ef8d58069cc6dc211977bc4a17b1e7411ce774e6c8c WatchSource:0}: Error finding container 4f1676d38524b8e20b003ef8d58069cc6dc211977bc4a17b1e7411ce774e6c8c: Status 404 returned error can't find the container with id 4f1676d38524b8e20b003ef8d58069cc6dc211977bc4a17b1e7411ce774e6c8c
Jul 08 16:59:14.895723 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.895687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" event={"ID":"e717f006-ba23-401e-9efe-422da3ea256e","Type":"ContainerStarted","Data":"e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2"}
Jul 08 16:59:14.895904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.895757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" event={"ID":"e717f006-ba23-401e-9efe-422da3ea256e","Type":"ContainerStarted","Data":"10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15"}
Jul 08 16:59:14.895904 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.895790    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" event={"ID":"e717f006-ba23-401e-9efe-422da3ea256e","Type":"ContainerStarted","Data":"4f1676d38524b8e20b003ef8d58069cc6dc211977bc4a17b1e7411ce774e6c8c"}
Jul 08 16:59:14.896020 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.895933    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:14.897487 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.897459    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" event={"ID":"099312d7-f64f-48a3-801e-18d3e9055d25","Type":"ContainerStarted","Data":"08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886"}
Jul 08 16:59:14.897617 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.897510    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:14.914793 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.914751    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podStartSLOduration=1.914738909 podStartE2EDuration="1.914738909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:59:14.913875716 +0000 UTC m=+1495.903432486" watchObservedRunningTime="2026-07-08 16:59:14.914738909 +0000 UTC m=+1495.904295652"
Jul 08 16:59:14.931444 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:14.931398    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podStartSLOduration=1.9313831879999999 podStartE2EDuration="1.931383188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:59:14.930987358 +0000 UTC m=+1495.920544138" watchObservedRunningTime="2026-07-08 16:59:14.931383188 +0000 UTC m=+1495.920939932"
Jul 08 16:59:15.544655 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:15.544619    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 08 16:59:15.545021 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:15.544619    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 08 16:59:15.901918 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:15.901887    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:15.902085 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:15.901948    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:15.903140 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:15.903111    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 08 16:59:15.903229 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:15.903111    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 08 16:59:16.161787 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.161735    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:59:16.286828 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.286798    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls\") pod \"eafd9458-b52a-488b-9c7a-804b22f35fed\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") "
Jul 08 16:59:16.286925 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.286848    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh8tf\" (UniqueName: \"kubernetes.io/projected/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-api-access-qh8tf\") pod \"eafd9458-b52a-488b-9c7a-804b22f35fed\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") "
Jul 08 16:59:16.286925 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.286878    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-rbac-proxy-sar-config\") pod \"eafd9458-b52a-488b-9c7a-804b22f35fed\" (UID: \"eafd9458-b52a-488b-9c7a-804b22f35fed\") "
Jul 08 16:59:16.287311 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.287288    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eafd9458-b52a-488b-9c7a-804b22f35fed" (UID: "eafd9458-b52a-488b-9c7a-804b22f35fed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:59:16.288889 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.288861    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eafd9458-b52a-488b-9c7a-804b22f35fed" (UID: "eafd9458-b52a-488b-9c7a-804b22f35fed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:59:16.288982 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.288929    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-api-access-qh8tf" (OuterVolumeSpecName: "kube-api-access-qh8tf") pod "eafd9458-b52a-488b-9c7a-804b22f35fed" (UID: "eafd9458-b52a-488b-9c7a-804b22f35fed"). InnerVolumeSpecName "kube-api-access-qh8tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:59:16.388158 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.388129    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafd9458-b52a-488b-9c7a-804b22f35fed-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:59:16.388158 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.388156    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh8tf\" (UniqueName: \"kubernetes.io/projected/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-api-access-qh8tf\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:59:16.388301 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.388166    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafd9458-b52a-488b-9c7a-804b22f35fed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:59:16.401112 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.401092    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:59:16.489381 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.489321    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-rbac-proxy-sar-config\") pod \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") "
Jul 08 16:59:16.489479 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.489438    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9bkj\" (UniqueName: \"kubernetes.io/projected/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-api-access-z9bkj\") pod \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") "
Jul 08 16:59:16.489479 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.489465    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-proxy-tls\") pod \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\" (UID: \"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6\") "
Jul 08 16:59:16.489774 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.489666    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" (UID: "dcd51f3a-9e78-4902-887f-fb6bcda9b5a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:59:16.491444 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.491426    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" (UID: "dcd51f3a-9e78-4902-887f-fb6bcda9b5a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:59:16.491507 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.491464    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-api-access-z9bkj" (OuterVolumeSpecName: "kube-api-access-z9bkj") pod "dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" (UID: "dcd51f3a-9e78-4902-887f-fb6bcda9b5a6"). InnerVolumeSpecName "kube-api-access-z9bkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:59:16.590521 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.590495    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9bkj\" (UniqueName: \"kubernetes.io/projected/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-api-access-z9bkj\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:59:16.590521 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.590522    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:59:16.590885 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.590537    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 16:59:16.907226 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.907198    2575 generic.go:356] "Generic (PLEG): container finished" podID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerID="b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1" exitCode=0
Jul 08 16:59:16.907459 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.907248    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"
Jul 08 16:59:16.907459 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.907297    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" event={"ID":"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6","Type":"ContainerDied","Data":"b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1"}
Jul 08 16:59:16.907459 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.907357    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk" event={"ID":"dcd51f3a-9e78-4902-887f-fb6bcda9b5a6","Type":"ContainerDied","Data":"6397b4f800c78492436e7fff2618c242b28546fe9afabc4ba67a6f0eb1a4e596"}
Jul 08 16:59:16.907459 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.907390    2575 scope.go:117] "RemoveContainer" containerID="452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3"
Jul 08 16:59:16.909180 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.909154    2575 generic.go:356] "Generic (PLEG): container finished" podID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerID="1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15" exitCode=0
Jul 08 16:59:16.909297 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.909215    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"
Jul 08 16:59:16.909297 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.909242    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" event={"ID":"eafd9458-b52a-488b-9c7a-804b22f35fed","Type":"ContainerDied","Data":"1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15"}
Jul 08 16:59:16.909404 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.909300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt" event={"ID":"eafd9458-b52a-488b-9c7a-804b22f35fed","Type":"ContainerDied","Data":"d73109c719db97ef2e23305b91f7117098a4b18b4c4e2ac8632f4a07ecf71612"}
Jul 08 16:59:16.909929 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.909895    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 08 16:59:16.910060 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.909903    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 08 16:59:16.916807 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.916765    2575 scope.go:117] "RemoveContainer" containerID="b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1"
Jul 08 16:59:16.924506 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.924490    2575 scope.go:117] "RemoveContainer" containerID="452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3"
Jul 08 16:59:16.924766 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:59:16.924748    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3\": container with ID starting with 452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3 not found: ID does not exist" containerID="452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3"
Jul 08 16:59:16.924837 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.924772    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3"} err="failed to get container status \"452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3\": rpc error: code = NotFound desc = could not find container \"452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3\": container with ID starting with 452f95713423a834732661cc659f3c502ee7f44a20151b813897d49850b97be3 not found: ID does not exist"
Jul 08 16:59:16.924837 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.924788    2575 scope.go:117] "RemoveContainer" containerID="b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1"
Jul 08 16:59:16.925020 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:59:16.925005    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1\": container with ID starting with b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1 not found: ID does not exist" containerID="b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1"
Jul 08 16:59:16.925079 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.925022    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1"} err="failed to get container status \"b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1\": rpc error: code = NotFound desc = could not find container \"b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1\": container with ID starting with b2cd4ed8b38062f23a25422dee9f55565b516b433a4d701bfd27643c11820ae1 not found: ID does not exist"
Jul 08 16:59:16.925079 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.925034    2575 scope.go:117] "RemoveContainer" containerID="ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db"
Jul 08 16:59:16.932352 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.932332    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"]
Jul 08 16:59:16.933693 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.933675    2575 scope.go:117] "RemoveContainer" containerID="1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15"
Jul 08 16:59:16.936381 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.936361    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-178bc-predictor-79b66c889d-d8dvk"]
Jul 08 16:59:16.940906 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.940890    2575 scope.go:117] "RemoveContainer" containerID="ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db"
Jul 08 16:59:16.941124 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:59:16.941107    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db\": container with ID starting with ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db not found: ID does not exist" containerID="ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db"
Jul 08 16:59:16.941181 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.941130    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db"} err="failed to get container status \"ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db\": rpc error: code = NotFound desc = could not find container \"ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db\": container with ID starting with ead996b6ee86abeb7ea5108753252660b4c2cbec46f12b84a1b7af104fad82db not found: ID does not exist"
Jul 08 16:59:16.941181 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.941146    2575 scope.go:117] "RemoveContainer" containerID="1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15"
Jul 08 16:59:16.941396 ip-10-0-135-143 kubenswrapper[2575]: E0708 16:59:16.941380    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15\": container with ID starting with 1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15 not found: ID does not exist" containerID="1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15"
Jul 08 16:59:16.941437 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.941402    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15"} err="failed to get container status \"1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15\": rpc error: code = NotFound desc = could not find container \"1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15\": container with ID starting with 1363e577ebf23e8ef4e7483e7986fbfd00012fe5ea8b872df76db646eb422e15 not found: ID does not exist"
Jul 08 16:59:16.946189 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.946170    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"]
Jul 08 16:59:16.949712 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:16.949693    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-178bc-predictor-f7f9d7fcf-nxwzt"]
Jul 08 16:59:17.608616 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:17.608585    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" path="/var/lib/kubelet/pods/dcd51f3a-9e78-4902-887f-fb6bcda9b5a6/volumes"
Jul 08 16:59:17.609025 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:17.609011    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" path="/var/lib/kubelet/pods/eafd9458-b52a-488b-9c7a-804b22f35fed/volumes"
Jul 08 16:59:19.532943 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:19.532914    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:59:19.533375 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:19.532950    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 16:59:21.915407 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:21.915361    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 16:59:21.916004 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:21.915799    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 16:59:21.916004 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:21.915933    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 08 16:59:21.916303 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:21.916279    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 08 16:59:31.265091 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:31.265060    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:31.916483 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:31.916440    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 08 16:59:31.916669 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:31.916440    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 08 16:59:41.916499 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:41.916412    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 08 16:59:41.916892 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:41.916413    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 08 16:59:51.916693 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:51.916656    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 08 16:59:51.917113 ip-10-0-135-143 kubenswrapper[2575]: I0708 16:59:51.916667    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 08 17:00:01.916696 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:00:01.916631    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 17:00:01.917152 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:00:01.916752    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 17:00:13.347377 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:00:13.347345    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:13.486972 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:00:13.486941    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:04:19.558329 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:04:19.558302    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:04:19.559506 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:04:19.559485    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:07:02.649082 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:02.649021    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:28.173206 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.173167    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:28.207769 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.207740    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"]
Jul 08 17:07:28.208002 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.207952    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container" containerID="cri-o://44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99" gracePeriod=30
Jul 08 17:07:28.208163 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.208011    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kube-rbac-proxy" containerID="cri-o://f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5" gracePeriod=30
Jul 08 17:07:28.255679 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.255636    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"]
Jul 08 17:07:28.255901 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.255872    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container" containerID="cri-o://82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219" gracePeriod=30
Jul 08 17:07:28.255901 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.255884    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kube-rbac-proxy" containerID="cri-o://14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b" gracePeriod=30
Jul 08 17:07:28.281100 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.281074    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:28.297480 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.297451    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"]
Jul 08 17:07:28.298013 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.297989    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kube-rbac-proxy"
Jul 08 17:07:28.298109 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298016    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kube-rbac-proxy"
Jul 08 17:07:28.298109 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298038    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container"
Jul 08 17:07:28.298109 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298047    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container"
Jul 08 17:07:28.298109 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298071    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container"
Jul 08 17:07:28.298109 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298080    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container"
Jul 08 17:07:28.298109 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298091    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kube-rbac-proxy"
Jul 08 17:07:28.298109 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298100    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kube-rbac-proxy"
Jul 08 17:07:28.298482 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298208    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kube-rbac-proxy"
Jul 08 17:07:28.298482 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298222    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kserve-container"
Jul 08 17:07:28.298482 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298235    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd51f3a-9e78-4902-887f-fb6bcda9b5a6" containerName="kube-rbac-proxy"
Jul 08 17:07:28.298482 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.298245    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafd9458-b52a-488b-9c7a-804b22f35fed" containerName="kserve-container"
Jul 08 17:07:28.301955 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.301933    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.304219 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.304198    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2c597-kube-rbac-proxy-sar-config\""
Jul 08 17:07:28.304321 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.304233    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2c597-predictor-serving-cert\""
Jul 08 17:07:28.311149 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.311125    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"]
Jul 08 17:07:28.337588 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.337561    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"]
Jul 08 17:07:28.341609 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.341584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.343881 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.343861    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2c597-predictor-serving-cert\""
Jul 08 17:07:28.344005 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.343861    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2c597-kube-rbac-proxy-sar-config\""
Jul 08 17:07:28.352325 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.352245    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"]
Jul 08 17:07:28.383492 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.383470    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:28.399912 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.399891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njmsh\" (UniqueName: \"kubernetes.io/projected/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-api-access-njmsh\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.400029 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.399948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-proxy-tls\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.400088 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.400046    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.501266 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.501180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njmsh\" (UniqueName: \"kubernetes.io/projected/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-api-access-njmsh\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.501266 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.501236    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzcl\" (UniqueName: \"kubernetes.io/projected/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-api-access-ztzcl\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.501266 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.501260    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-proxy-tls\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.501507 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.501284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-proxy-tls\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.501507 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.501308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.501507 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.501332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.501966 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.501939    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.503893 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.503867    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-proxy-tls\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.512712 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.512690    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njmsh\" (UniqueName: \"kubernetes.io/projected/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-api-access-njmsh\") pod \"success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") " pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.523738 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.523711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:28.602022 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.601987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzcl\" (UniqueName: \"kubernetes.io/projected/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-api-access-ztzcl\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.602210 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.602039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-proxy-tls\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.602210 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.602066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.602761 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.602738    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.604504 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.604483    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-proxy-tls\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.612364 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.612340    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:28.613984 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.613966    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzcl\" (UniqueName: \"kubernetes.io/projected/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-api-access-ztzcl\") pod \"error-404-isvc-2c597-predictor-6bdcb48c76-7578d\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") " pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.653791 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.653752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:28.755512 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.755487    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"]
Jul 08 17:07:28.757422 ip-10-0-135-143 kubenswrapper[2575]: W0708 17:07:28.757378    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2204dcf_bfa9_4818_8fe4_ba80bb0ab705.slice/crio-69843e471f8a782639c85d14980f19a485b864909ffd9b1ee66ebeb43fc88f57 WatchSource:0}: Error finding container 69843e471f8a782639c85d14980f19a485b864909ffd9b1ee66ebeb43fc88f57: Status 404 returned error can't find the container with id 69843e471f8a782639c85d14980f19a485b864909ffd9b1ee66ebeb43fc88f57
Jul 08 17:07:28.802591 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:28.802569    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"]
Jul 08 17:07:28.805437 ip-10-0-135-143 kubenswrapper[2575]: W0708 17:07:28.805412    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e911cc7_1bab_45a3_95a8_d6b37031c1c7.slice/crio-c714155fd71869be4cd96079973cc31779b6806c080efdbfcb4397b5bb54da48 WatchSource:0}: Error finding container c714155fd71869be4cd96079973cc31779b6806c080efdbfcb4397b5bb54da48: Status 404 returned error can't find the container with id c714155fd71869be4cd96079973cc31779b6806c080efdbfcb4397b5bb54da48
Jul 08 17:07:29.015230 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.015147    2575 generic.go:356] "Generic (PLEG): container finished" podID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerID="14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b" exitCode=2
Jul 08 17:07:29.015230 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.015216    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" event={"ID":"d176335b-e0f2-4943-8d7d-4bd699d52418","Type":"ContainerDied","Data":"14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b"}
Jul 08 17:07:29.016902 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.016869    2575 generic.go:356] "Generic (PLEG): container finished" podID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerID="f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5" exitCode=2
Jul 08 17:07:29.017012 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.016909    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" event={"ID":"16b45dea-ae5b-4b61-9b05-209978bc4a1f","Type":"ContainerDied","Data":"f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5"}
Jul 08 17:07:29.018720 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.018695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" event={"ID":"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705","Type":"ContainerStarted","Data":"de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3"}
Jul 08 17:07:29.018824 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.018745    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" event={"ID":"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705","Type":"ContainerStarted","Data":"4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9"}
Jul 08 17:07:29.018824 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.018767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" event={"ID":"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705","Type":"ContainerStarted","Data":"69843e471f8a782639c85d14980f19a485b864909ffd9b1ee66ebeb43fc88f57"}
Jul 08 17:07:29.018928 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.018890    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:29.020300 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.020280    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" event={"ID":"4e911cc7-1bab-45a3-95a8-d6b37031c1c7","Type":"ContainerStarted","Data":"5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee"}
Jul 08 17:07:29.020393 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.020318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" event={"ID":"4e911cc7-1bab-45a3-95a8-d6b37031c1c7","Type":"ContainerStarted","Data":"2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805"}
Jul 08 17:07:29.020393 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.020350    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:29.020393 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.020370    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" event={"ID":"4e911cc7-1bab-45a3-95a8-d6b37031c1c7","Type":"ContainerStarted","Data":"c714155fd71869be4cd96079973cc31779b6806c080efdbfcb4397b5bb54da48"}
Jul 08 17:07:29.020498 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.020400    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:29.022026 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.022000    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 08 17:07:29.037805 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.037763    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podStartSLOduration=1.037751234 podStartE2EDuration="1.037751234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:07:29.035936518 +0000 UTC m=+1990.025493261" watchObservedRunningTime="2026-07-08 17:07:29.037751234 +0000 UTC m=+1990.027307981"
Jul 08 17:07:29.053572 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.053532    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podStartSLOduration=1.053520273 podStartE2EDuration="1.053520273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:07:29.052945494 +0000 UTC m=+1990.042502237" watchObservedRunningTime="2026-07-08 17:07:29.053520273 +0000 UTC m=+1990.043077090"
Jul 08 17:07:29.311193 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:29.311111    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:30.025365 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:30.025332    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:30.025365 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:30.025355    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 08 17:07:30.026610 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:30.026588    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 08 17:07:30.609155 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:30.609110    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 08 17:07:30.615942 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:30.615910    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 08 17:07:31.029174 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.029091    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 08 17:07:31.459202 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.459172    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:31.552216 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.552194    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 17:07:31.667533 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.667509    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 17:07:31.729209 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.729179    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls\") pod \"d176335b-e0f2-4943-8d7d-4bd699d52418\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") "
Jul 08 17:07:31.729368 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.729224    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-rbac-proxy-sar-config\") pod \"d176335b-e0f2-4943-8d7d-4bd699d52418\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") "
Jul 08 17:07:31.729368 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.729256    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzxvj\" (UniqueName: \"kubernetes.io/projected/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-api-access-hzxvj\") pod \"d176335b-e0f2-4943-8d7d-4bd699d52418\" (UID: \"d176335b-e0f2-4943-8d7d-4bd699d52418\") "
Jul 08 17:07:31.729574 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.729552    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d176335b-e0f2-4943-8d7d-4bd699d52418" (UID: "d176335b-e0f2-4943-8d7d-4bd699d52418"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:07:31.731303 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.731282    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d176335b-e0f2-4943-8d7d-4bd699d52418" (UID: "d176335b-e0f2-4943-8d7d-4bd699d52418"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:07:31.731368 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.731334    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-api-access-hzxvj" (OuterVolumeSpecName: "kube-api-access-hzxvj") pod "d176335b-e0f2-4943-8d7d-4bd699d52418" (UID: "d176335b-e0f2-4943-8d7d-4bd699d52418"). InnerVolumeSpecName "kube-api-access-hzxvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:07:31.830797 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.830710    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-rbac-proxy-sar-config\") pod \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") "
Jul 08 17:07:31.830797 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.830752    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b45dea-ae5b-4b61-9b05-209978bc4a1f-proxy-tls\") pod \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") "
Jul 08 17:07:31.830797 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.830772    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w89zv\" (UniqueName: \"kubernetes.io/projected/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-api-access-w89zv\") pod \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\" (UID: \"16b45dea-ae5b-4b61-9b05-209978bc4a1f\") "
Jul 08 17:07:31.831012 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.830980    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d176335b-e0f2-4943-8d7d-4bd699d52418-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:07:31.831012 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.830998    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:07:31.831079 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.831013    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxvj\" (UniqueName: \"kubernetes.io/projected/d176335b-e0f2-4943-8d7d-4bd699d52418-kube-api-access-hzxvj\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:07:31.831079 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.831064    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "16b45dea-ae5b-4b61-9b05-209978bc4a1f" (UID: "16b45dea-ae5b-4b61-9b05-209978bc4a1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:07:31.832882 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.832858    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-api-access-w89zv" (OuterVolumeSpecName: "kube-api-access-w89zv") pod "16b45dea-ae5b-4b61-9b05-209978bc4a1f" (UID: "16b45dea-ae5b-4b61-9b05-209978bc4a1f"). InnerVolumeSpecName "kube-api-access-w89zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:07:31.832882 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.832870    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b45dea-ae5b-4b61-9b05-209978bc4a1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16b45dea-ae5b-4b61-9b05-209978bc4a1f" (UID: "16b45dea-ae5b-4b61-9b05-209978bc4a1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:07:31.931790 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.931759    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:07:31.931790 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.931784    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b45dea-ae5b-4b61-9b05-209978bc4a1f-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:07:31.931790 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:31.931794    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w89zv\" (UniqueName: \"kubernetes.io/projected/16b45dea-ae5b-4b61-9b05-209978bc4a1f-kube-api-access-w89zv\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:07:32.034980 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.034941    2575 generic.go:356] "Generic (PLEG): container finished" podID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerID="82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219" exitCode=0
Jul 08 17:07:32.035173 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.034992    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"
Jul 08 17:07:32.035173 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.035033    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" event={"ID":"d176335b-e0f2-4943-8d7d-4bd699d52418","Type":"ContainerDied","Data":"82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219"}
Jul 08 17:07:32.035173 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.035109    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9" event={"ID":"d176335b-e0f2-4943-8d7d-4bd699d52418","Type":"ContainerDied","Data":"3542b7e6cd9d3303061e34af7f48044901a4af0a763aea4407c6812510bfda0d"}
Jul 08 17:07:32.035173 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.035145    2575 scope.go:117] "RemoveContainer" containerID="14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b"
Jul 08 17:07:32.036664 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.036627    2575 generic.go:356] "Generic (PLEG): container finished" podID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerID="44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99" exitCode=0
Jul 08 17:07:32.036787 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.036682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" event={"ID":"16b45dea-ae5b-4b61-9b05-209978bc4a1f","Type":"ContainerDied","Data":"44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99"}
Jul 08 17:07:32.036787 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.036689    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"
Jul 08 17:07:32.036787 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.036731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p" event={"ID":"16b45dea-ae5b-4b61-9b05-209978bc4a1f","Type":"ContainerDied","Data":"f1454533baccddbf9ee546243ae67954c5b21067952b863f1c57668164a74a55"}
Jul 08 17:07:32.044722 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.044704    2575 scope.go:117] "RemoveContainer" containerID="82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219"
Jul 08 17:07:32.052202 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.052185    2575 scope.go:117] "RemoveContainer" containerID="14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b"
Jul 08 17:07:32.052459 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:07:32.052433    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b\": container with ID starting with 14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b not found: ID does not exist" containerID="14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b"
Jul 08 17:07:32.052544 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.052459    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b"} err="failed to get container status \"14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b\": rpc error: code = NotFound desc = could not find container \"14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b\": container with ID starting with 14e4b081ac7e0760ebe715afb7a7cedfe56aeb71470329a1a0eb9a65267cd05b not found: ID does not exist"
Jul 08 17:07:32.052544 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.052475    2575 scope.go:117] "RemoveContainer" containerID="82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219"
Jul 08 17:07:32.052721 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:07:32.052699    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219\": container with ID starting with 82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219 not found: ID does not exist" containerID="82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219"
Jul 08 17:07:32.052763 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.052730    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219"} err="failed to get container status \"82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219\": rpc error: code = NotFound desc = could not find container \"82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219\": container with ID starting with 82f735fe9c7cb181b50adcde669260bb1d0c1dde25db2a51ed8f6c06e8956219 not found: ID does not exist"
Jul 08 17:07:32.052763 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.052753    2575 scope.go:117] "RemoveContainer" containerID="f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5"
Jul 08 17:07:32.064422 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.064383    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"]
Jul 08 17:07:32.065934 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.065916    2575 scope.go:117] "RemoveContainer" containerID="44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99"
Jul 08 17:07:32.070125 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.070106    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-806a8-predictor-d7dcd8679-j829p"]
Jul 08 17:07:32.075142 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.075126    2575 scope.go:117] "RemoveContainer" containerID="f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5"
Jul 08 17:07:32.075450 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:07:32.075425    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5\": container with ID starting with f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5 not found: ID does not exist" containerID="f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5"
Jul 08 17:07:32.075529 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.075475    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5"} err="failed to get container status \"f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5\": rpc error: code = NotFound desc = could not find container \"f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5\": container with ID starting with f20b36ed06e42a73b505d64b3f27087aef815c044908c789aac2ad0c75527ef5 not found: ID does not exist"
Jul 08 17:07:32.075529 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.075511    2575 scope.go:117] "RemoveContainer" containerID="44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99"
Jul 08 17:07:32.075920 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:07:32.075891    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99\": container with ID starting with 44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99 not found: ID does not exist" containerID="44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99"
Jul 08 17:07:32.076001 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.075931    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99"} err="failed to get container status \"44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99\": rpc error: code = NotFound desc = could not find container \"44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99\": container with ID starting with 44d74d9f0f5531cdd5eeffdd69cd38331658fa066a50d256c7c8eab7dbbcae99 not found: ID does not exist"
Jul 08 17:07:32.080535 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.080511    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"]
Jul 08 17:07:32.084438 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:32.084389    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-806a8-predictor-5595d45646-t22t9"]
Jul 08 17:07:33.609169 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:33.609137    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" path="/var/lib/kubelet/pods/16b45dea-ae5b-4b61-9b05-209978bc4a1f/volumes"
Jul 08 17:07:33.609656 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:33.609532    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" path="/var/lib/kubelet/pods/d176335b-e0f2-4943-8d7d-4bd699d52418/volumes"
Jul 08 17:07:35.029929 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:35.029899    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:07:35.030387 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:35.030364    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 08 17:07:36.033174 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:36.033146    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:07:36.033633 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:36.033606    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 08 17:07:45.030837 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:45.030792    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 08 17:07:46.033656 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:46.033611    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 08 17:07:55.030769 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:55.030724    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 08 17:07:56.033576 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:07:56.033539    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 08 17:08:05.031197 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:05.031158    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 08 17:08:06.033635 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:06.033593    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 08 17:08:15.032211 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:15.032180    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:08:16.035259 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:16.035225    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:08:27.870416 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:27.870371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:27.951186 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:27.951154    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"]
Jul 08 17:08:27.951434 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:27.951408    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container" containerID="cri-o://10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15" gracePeriod=30
Jul 08 17:08:27.951554 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:27.951495    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kube-rbac-proxy" containerID="cri-o://e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2" gracePeriod=30
Jul 08 17:08:27.974747 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:27.974717    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:28.012573 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.012542    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"]
Jul 08 17:08:28.012806 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.012783    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container" containerID="cri-o://e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb" gracePeriod=30
Jul 08 17:08:28.012868 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.012824    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kube-rbac-proxy" containerID="cri-o://08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886" gracePeriod=30
Jul 08 17:08:28.028487 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028465    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"]
Jul 08 17:08:28.028860 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028830    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kube-rbac-proxy"
Jul 08 17:08:28.028860 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028850    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kube-rbac-proxy"
Jul 08 17:08:28.029013 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028863    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container"
Jul 08 17:08:28.029013 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028872    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container"
Jul 08 17:08:28.029013 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028883    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kube-rbac-proxy"
Jul 08 17:08:28.029013 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028892    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kube-rbac-proxy"
Jul 08 17:08:28.029013 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028939    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container"
Jul 08 17:08:28.029013 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.028948    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container"
Jul 08 17:08:28.029690 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.029040    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kube-rbac-proxy"
Jul 08 17:08:28.029690 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.029049    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kube-rbac-proxy"
Jul 08 17:08:28.029690 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.029065    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d176335b-e0f2-4943-8d7d-4bd699d52418" containerName="kserve-container"
Jul 08 17:08:28.029690 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.029082    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16b45dea-ae5b-4b61-9b05-209978bc4a1f" containerName="kserve-container"
Jul 08 17:08:28.032830 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.032814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.035119 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.035100    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d09b9-predictor-serving-cert\""
Jul 08 17:08:28.035370 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.035354    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d09b9-kube-rbac-proxy-sar-config\""
Jul 08 17:08:28.042710 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.042688    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"]
Jul 08 17:08:28.091126 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.091101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:28.120305 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.120282    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"]
Jul 08 17:08:28.123623 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.123574    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.126268 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.126246    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d09b9-kube-rbac-proxy-sar-config\""
Jul 08 17:08:28.126367 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.126286    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d09b9-predictor-serving-cert\""
Jul 08 17:08:28.132278 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.132255    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"]
Jul 08 17:08:28.172901 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.172874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.173024 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.172933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.173024 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.172951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwm8q\" (UniqueName: \"kubernetes.io/projected/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-api-access-wwm8q\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.197368 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.197340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:28.273566 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.273545    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthdb\" (UniqueName: \"kubernetes.io/projected/86378337-6f02-4b24-b099-66c502100f55-kube-api-access-jthdb\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.273721 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.273573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86378337-6f02-4b24-b099-66c502100f55-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.273721 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.273594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86378337-6f02-4b24-b099-66c502100f55-proxy-tls\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.273721 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.273627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.273721 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.273693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwm8q\" (UniqueName: \"kubernetes.io/projected/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-api-access-wwm8q\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.273929 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.273911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.274276 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.274254    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.276204 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.276183    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.286196 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.286171    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwm8q\" (UniqueName: \"kubernetes.io/projected/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-api-access-wwm8q\") pod \"success-200-isvc-d09b9-predictor-75cd6867df-q6mgb\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") " pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.287895 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.287871    2575 generic.go:356] "Generic (PLEG): container finished" podID="099312d7-f64f-48a3-801e-18d3e9055d25" containerID="08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886" exitCode=2
Jul 08 17:08:28.287996 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.287944    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" event={"ID":"099312d7-f64f-48a3-801e-18d3e9055d25","Type":"ContainerDied","Data":"08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886"}
Jul 08 17:08:28.289841 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.289820    2575 generic.go:356] "Generic (PLEG): container finished" podID="e717f006-ba23-401e-9efe-422da3ea256e" containerID="e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2" exitCode=2
Jul 08 17:08:28.289942 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.289876    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" event={"ID":"e717f006-ba23-401e-9efe-422da3ea256e","Type":"ContainerDied","Data":"e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2"}
Jul 08 17:08:28.344662 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.344622    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:28.375096 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.375041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthdb\" (UniqueName: \"kubernetes.io/projected/86378337-6f02-4b24-b099-66c502100f55-kube-api-access-jthdb\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.375096 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.375074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86378337-6f02-4b24-b099-66c502100f55-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.375096 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.375096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86378337-6f02-4b24-b099-66c502100f55-proxy-tls\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.375663 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.375628    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86378337-6f02-4b24-b099-66c502100f55-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.377418 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.377393    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86378337-6f02-4b24-b099-66c502100f55-proxy-tls\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.386876 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.386854    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthdb\" (UniqueName: \"kubernetes.io/projected/86378337-6f02-4b24-b099-66c502100f55-kube-api-access-jthdb\") pod \"error-404-isvc-d09b9-predictor-6cb94d46cb-489qc\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") " pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.395534 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.395283    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:28.434633 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.434586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:28.499256 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.492889    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"]
Jul 08 17:08:28.499256 ip-10-0-135-143 kubenswrapper[2575]: W0708 17:08:28.494520    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7c6494_5c28_4353_8f42_620c8e8d277b.slice/crio-c0e477dfb5d12e63321e0a3d6e46e0029c6bb7d0e93aaf45c71690539e02a147 WatchSource:0}: Error finding container c0e477dfb5d12e63321e0a3d6e46e0029c6bb7d0e93aaf45c71690539e02a147: Status 404 returned error can't find the container with id c0e477dfb5d12e63321e0a3d6e46e0029c6bb7d0e93aaf45c71690539e02a147
Jul 08 17:08:28.575837 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.575812    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"]
Jul 08 17:08:28.584277 ip-10-0-135-143 kubenswrapper[2575]: W0708 17:08:28.584043    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86378337_6f02_4b24_b099_66c502100f55.slice/crio-30335bdfe00de4193d3a196f5bb0d1ae46a48b33889017e649ec4afbf8525927 WatchSource:0}: Error finding container 30335bdfe00de4193d3a196f5bb0d1ae46a48b33889017e649ec4afbf8525927: Status 404 returned error can't find the container with id 30335bdfe00de4193d3a196f5bb0d1ae46a48b33889017e649ec4afbf8525927
Jul 08 17:08:28.799823 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:28.799780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:29.295309 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.295273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" event={"ID":"7f7c6494-5c28-4353-8f42-620c8e8d277b","Type":"ContainerStarted","Data":"302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a"}
Jul 08 17:08:29.295815 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.295340    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" event={"ID":"7f7c6494-5c28-4353-8f42-620c8e8d277b","Type":"ContainerStarted","Data":"58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7"}
Jul 08 17:08:29.295815 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.295378    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:29.295815 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.295410    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:29.295815 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.295437    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" event={"ID":"7f7c6494-5c28-4353-8f42-620c8e8d277b","Type":"ContainerStarted","Data":"c0e477dfb5d12e63321e0a3d6e46e0029c6bb7d0e93aaf45c71690539e02a147"}
Jul 08 17:08:29.297285 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.297255    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 08 17:08:29.297405 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.297300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" event={"ID":"86378337-6f02-4b24-b099-66c502100f55","Type":"ContainerStarted","Data":"11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff"}
Jul 08 17:08:29.297405 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.297341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" event={"ID":"86378337-6f02-4b24-b099-66c502100f55","Type":"ContainerStarted","Data":"8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8"}
Jul 08 17:08:29.297405 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.297369    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" event={"ID":"86378337-6f02-4b24-b099-66c502100f55","Type":"ContainerStarted","Data":"30335bdfe00de4193d3a196f5bb0d1ae46a48b33889017e649ec4afbf8525927"}
Jul 08 17:08:29.297405 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.297400    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:29.297547 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.297467    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:29.298576 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.298555    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 08 17:08:29.316433 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.316368    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podStartSLOduration=1.316353038 podStartE2EDuration="1.316353038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:08:29.315360198 +0000 UTC m=+2050.304916960" watchObservedRunningTime="2026-07-08 17:08:29.316353038 +0000 UTC m=+2050.305909782"
Jul 08 17:08:29.337573 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:29.337531    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podStartSLOduration=1.337520182 podStartE2EDuration="1.337520182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:08:29.336466476 +0000 UTC m=+2050.326023220" watchObservedRunningTime="2026-07-08 17:08:29.337520182 +0000 UTC m=+2050.327076924"
Jul 08 17:08:30.302462 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:30.302426    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 08 17:08:30.302899 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:30.302510    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 08 17:08:31.222536 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.222511    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 17:08:31.225229 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.225210    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 17:08:31.301781 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.301708    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099312d7-f64f-48a3-801e-18d3e9055d25-kube-rbac-proxy-sar-config\") pod \"099312d7-f64f-48a3-801e-18d3e9055d25\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") "
Jul 08 17:08:31.301781 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.301759    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv578\" (UniqueName: \"kubernetes.io/projected/099312d7-f64f-48a3-801e-18d3e9055d25-kube-api-access-hv578\") pod \"099312d7-f64f-48a3-801e-18d3e9055d25\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") "
Jul 08 17:08:31.301948 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.301795    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099312d7-f64f-48a3-801e-18d3e9055d25-proxy-tls\") pod \"099312d7-f64f-48a3-801e-18d3e9055d25\" (UID: \"099312d7-f64f-48a3-801e-18d3e9055d25\") "
Jul 08 17:08:31.302072 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.302050    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099312d7-f64f-48a3-801e-18d3e9055d25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "099312d7-f64f-48a3-801e-18d3e9055d25" (UID: "099312d7-f64f-48a3-801e-18d3e9055d25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:08:31.303922 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.303892    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099312d7-f64f-48a3-801e-18d3e9055d25-kube-api-access-hv578" (OuterVolumeSpecName: "kube-api-access-hv578") pod "099312d7-f64f-48a3-801e-18d3e9055d25" (UID: "099312d7-f64f-48a3-801e-18d3e9055d25"). InnerVolumeSpecName "kube-api-access-hv578". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:08:31.304276 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.304085    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099312d7-f64f-48a3-801e-18d3e9055d25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "099312d7-f64f-48a3-801e-18d3e9055d25" (UID: "099312d7-f64f-48a3-801e-18d3e9055d25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:08:31.307713 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.307689    2575 generic.go:356] "Generic (PLEG): container finished" podID="e717f006-ba23-401e-9efe-422da3ea256e" containerID="10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15" exitCode=0
Jul 08 17:08:31.307826 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.307745    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"
Jul 08 17:08:31.307826 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.307767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" event={"ID":"e717f006-ba23-401e-9efe-422da3ea256e","Type":"ContainerDied","Data":"10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15"}
Jul 08 17:08:31.307939 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.307841    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c" event={"ID":"e717f006-ba23-401e-9efe-422da3ea256e","Type":"ContainerDied","Data":"4f1676d38524b8e20b003ef8d58069cc6dc211977bc4a17b1e7411ce774e6c8c"}
Jul 08 17:08:31.307939 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.307882    2575 scope.go:117] "RemoveContainer" containerID="e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2"
Jul 08 17:08:31.309285 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.309265    2575 generic.go:356] "Generic (PLEG): container finished" podID="099312d7-f64f-48a3-801e-18d3e9055d25" containerID="e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb" exitCode=0
Jul 08 17:08:31.309383 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.309320    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"
Jul 08 17:08:31.309383 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.309347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" event={"ID":"099312d7-f64f-48a3-801e-18d3e9055d25","Type":"ContainerDied","Data":"e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb"}
Jul 08 17:08:31.309483 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.309404    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79" event={"ID":"099312d7-f64f-48a3-801e-18d3e9055d25","Type":"ContainerDied","Data":"bc141e1f2b6d3ee50b2b13ae43d7f2567accec4dcff1031c967783e7d294570a"}
Jul 08 17:08:31.317349 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.317330    2575 scope.go:117] "RemoveContainer" containerID="10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15"
Jul 08 17:08:31.324671 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.324637    2575 scope.go:117] "RemoveContainer" containerID="e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2"
Jul 08 17:08:31.324904 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:31.324888    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2\": container with ID starting with e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2 not found: ID does not exist" containerID="e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2"
Jul 08 17:08:31.324976 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.324914    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2"} err="failed to get container status \"e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2\": rpc error: code = NotFound desc = could not find container \"e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2\": container with ID starting with e913bace216de1d73172129e1b5100cfaa3a8f54b95c299b1d75c8f7bcca42e2 not found: ID does not exist"
Jul 08 17:08:31.324976 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.324936    2575 scope.go:117] "RemoveContainer" containerID="10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15"
Jul 08 17:08:31.325188 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:31.325170    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15\": container with ID starting with 10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15 not found: ID does not exist" containerID="10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15"
Jul 08 17:08:31.325230 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.325195    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15"} err="failed to get container status \"10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15\": rpc error: code = NotFound desc = could not find container \"10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15\": container with ID starting with 10015b5fe7c9965858713054eaf40f9bffe2d479a4d95be0bec20f4ba612de15 not found: ID does not exist"
Jul 08 17:08:31.325230 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.325209    2575 scope.go:117] "RemoveContainer" containerID="08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886"
Jul 08 17:08:31.332040 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.332021    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"]
Jul 08 17:08:31.332421 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.332407    2575 scope.go:117] "RemoveContainer" containerID="e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb"
Jul 08 17:08:31.336874 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.336854    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2f8c-predictor-7c69fd84bb-cxh79"]
Jul 08 17:08:31.340069 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.340054    2575 scope.go:117] "RemoveContainer" containerID="08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886"
Jul 08 17:08:31.340312 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:31.340296    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886\": container with ID starting with 08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886 not found: ID does not exist" containerID="08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886"
Jul 08 17:08:31.340389 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.340319    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886"} err="failed to get container status \"08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886\": rpc error: code = NotFound desc = could not find container \"08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886\": container with ID starting with 08d67ae2765d1341b0a94b80fb89701bd43687482809d94d19dfb80e71fb0886 not found: ID does not exist"
Jul 08 17:08:31.340389 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.340341    2575 scope.go:117] "RemoveContainer" containerID="e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb"
Jul 08 17:08:31.340558 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:31.340541    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb\": container with ID starting with e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb not found: ID does not exist" containerID="e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb"
Jul 08 17:08:31.340611 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.340563    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb"} err="failed to get container status \"e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb\": rpc error: code = NotFound desc = could not find container \"e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb\": container with ID starting with e8a4e8cd89c8e3ed11a3567a30b7e4f207b9d434da04fa3f8cfd70473b7c66bb not found: ID does not exist"
Jul 08 17:08:31.403129 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.403100    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e717f006-ba23-401e-9efe-422da3ea256e-kube-rbac-proxy-sar-config\") pod \"e717f006-ba23-401e-9efe-422da3ea256e\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") "
Jul 08 17:08:31.403252 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.403164    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls\") pod \"e717f006-ba23-401e-9efe-422da3ea256e\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") "
Jul 08 17:08:31.403252 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.403225    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nl7r\" (UniqueName: \"kubernetes.io/projected/e717f006-ba23-401e-9efe-422da3ea256e-kube-api-access-8nl7r\") pod \"e717f006-ba23-401e-9efe-422da3ea256e\" (UID: \"e717f006-ba23-401e-9efe-422da3ea256e\") "
Jul 08 17:08:31.403373 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.403360    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099312d7-f64f-48a3-801e-18d3e9055d25-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:31.403432 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.403377    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099312d7-f64f-48a3-801e-18d3e9055d25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:31.403432 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.403387    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv578\" (UniqueName: \"kubernetes.io/projected/099312d7-f64f-48a3-801e-18d3e9055d25-kube-api-access-hv578\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:31.403432 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.403410    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e717f006-ba23-401e-9efe-422da3ea256e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e717f006-ba23-401e-9efe-422da3ea256e" (UID: "e717f006-ba23-401e-9efe-422da3ea256e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:08:31.405136 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.405117    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e717f006-ba23-401e-9efe-422da3ea256e-kube-api-access-8nl7r" (OuterVolumeSpecName: "kube-api-access-8nl7r") pod "e717f006-ba23-401e-9efe-422da3ea256e" (UID: "e717f006-ba23-401e-9efe-422da3ea256e"). InnerVolumeSpecName "kube-api-access-8nl7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:08:31.405194 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.405137    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e717f006-ba23-401e-9efe-422da3ea256e" (UID: "e717f006-ba23-401e-9efe-422da3ea256e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:08:31.504707 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.504681    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nl7r\" (UniqueName: \"kubernetes.io/projected/e717f006-ba23-401e-9efe-422da3ea256e-kube-api-access-8nl7r\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:31.504707 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.504704    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e717f006-ba23-401e-9efe-422da3ea256e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:31.504840 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.504715    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e717f006-ba23-401e-9efe-422da3ea256e-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:31.608249 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.608225    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" path="/var/lib/kubelet/pods/099312d7-f64f-48a3-801e-18d3e9055d25/volumes"
Jul 08 17:08:31.632754 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.632732    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"]
Jul 08 17:08:31.653664 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:31.638337    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2f8c-predictor-867ddf48ff-z8f8c"]
Jul 08 17:08:33.608921 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:33.608890    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e717f006-ba23-401e-9efe-422da3ea256e" path="/var/lib/kubelet/pods/e717f006-ba23-401e-9efe-422da3ea256e/volumes"
Jul 08 17:08:35.308150 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:35.308120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:08:35.308640 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:35.308553    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:08:35.308738 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:35.308688    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 08 17:08:35.309046 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:35.309016    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 08 17:08:38.457665 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.457613    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:38.528065 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.528034    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"]
Jul 08 17:08:38.528275 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.528251    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container" containerID="cri-o://2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805" gracePeriod=30
Jul 08 17:08:38.528352 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.528292    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kube-rbac-proxy" containerID="cri-o://5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee" gracePeriod=30
Jul 08 17:08:38.571806 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.571777    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:38.598946 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.598917    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"]
Jul 08 17:08:38.599148 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.599124    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container" containerID="cri-o://4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9" gracePeriod=30
Jul 08 17:08:38.599216 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.599151    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kube-rbac-proxy" containerID="cri-o://de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3" gracePeriod=30
Jul 08 17:08:38.611205 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611185    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"]
Jul 08 17:08:38.611515 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611504    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container"
Jul 08 17:08:38.611561 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611517    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container"
Jul 08 17:08:38.611561 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611530    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container"
Jul 08 17:08:38.611561 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611536    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container"
Jul 08 17:08:38.611561 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611548    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kube-rbac-proxy"
Jul 08 17:08:38.611561 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611554    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kube-rbac-proxy"
Jul 08 17:08:38.611734 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611565    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kube-rbac-proxy"
Jul 08 17:08:38.611734 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611570    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kube-rbac-proxy"
Jul 08 17:08:38.611734 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611616    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kube-rbac-proxy"
Jul 08 17:08:38.611734 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611623    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="099312d7-f64f-48a3-801e-18d3e9055d25" containerName="kserve-container"
Jul 08 17:08:38.611734 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611630    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kserve-container"
Jul 08 17:08:38.611734 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.611638    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e717f006-ba23-401e-9efe-422da3ea256e" containerName="kube-rbac-proxy"
Jul 08 17:08:38.616240 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.616223    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.618461 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.618402    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ab489-predictor-serving-cert\""
Jul 08 17:08:38.618621 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.618431    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ab489-kube-rbac-proxy-sar-config\""
Jul 08 17:08:38.626235 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.626206    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"]
Jul 08 17:08:38.654381 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.654353    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:38.757250 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.757221    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"]
Jul 08 17:08:38.760536 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.760520    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.763878 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.763856    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ab489-predictor-serving-cert\""
Jul 08 17:08:38.763986 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.763915    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ab489-kube-rbac-proxy-sar-config\""
Jul 08 17:08:38.770110 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.770092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.770183 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.770139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.770268 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.770249    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9nx\" (UniqueName: \"kubernetes.io/projected/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-api-access-5r9nx\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.778292 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.778268    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"]
Jul 08 17:08:38.808955 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.808927    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:38.871038 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.871008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.871179 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.871076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.871179 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.871133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9nx\" (UniqueName: \"kubernetes.io/projected/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-api-access-5r9nx\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.871179 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.871175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-proxy-tls\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.871330 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:38.871201    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ab489-predictor-serving-cert: secret "success-200-isvc-ab489-predictor-serving-cert" not found
Jul 08 17:08:38.871330 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.871209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4g42\" (UniqueName: \"kubernetes.io/projected/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-api-access-l4g42\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.871330 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:38.871272    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls podName:57dc77a7-b03c-4dad-a43b-96d7981d9400 nodeName:}" failed. No retries permitted until 2026-07-08 17:08:39.371249722 +0000 UTC m=+2060.360806443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls") pod "success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" (UID: "57dc77a7-b03c-4dad-a43b-96d7981d9400") : secret "success-200-isvc-ab489-predictor-serving-cert" not found
Jul 08 17:08:38.871330 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.871298    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.871682 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.871664    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.888092 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.888067    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9nx\" (UniqueName: \"kubernetes.io/projected/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-api-access-5r9nx\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:38.972220 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.972126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-proxy-tls\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.972220 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.972162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4g42\" (UniqueName: \"kubernetes.io/projected/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-api-access-l4g42\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.972220 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.972182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.972847 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.972827    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.974436 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.974414    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-proxy-tls\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:38.988709 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.988681    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:38.991913 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:38.991893    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4g42\" (UniqueName: \"kubernetes.io/projected/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-api-access-l4g42\") pod \"error-404-isvc-ab489-predictor-76897c7688-74fs9\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") " pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:39.069969 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.069945    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:39.093527 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.093480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:08:39.204039 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.204013    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"]
Jul 08 17:08:39.205969 ip-10-0-135-143 kubenswrapper[2575]: W0708 17:08:39.205943    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2ba3ac_69f2_47cb_937b_a54278f1ed13.slice/crio-c8763f26dea695974157bf82463199aca50ce612990f0522e2470b2cf5de8c61 WatchSource:0}: Error finding container c8763f26dea695974157bf82463199aca50ce612990f0522e2470b2cf5de8c61: Status 404 returned error can't find the container with id c8763f26dea695974157bf82463199aca50ce612990f0522e2470b2cf5de8c61
Jul 08 17:08:39.349934 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.349909    2575 generic.go:356] "Generic (PLEG): container finished" podID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerID="de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3" exitCode=2
Jul 08 17:08:39.350034 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.349985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" event={"ID":"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705","Type":"ContainerDied","Data":"de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3"}
Jul 08 17:08:39.351594 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.351573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" event={"ID":"2c2ba3ac-69f2-47cb-937b-a54278f1ed13","Type":"ContainerStarted","Data":"69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f"}
Jul 08 17:08:39.351751 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.351637    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:39.351751 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.351696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" event={"ID":"2c2ba3ac-69f2-47cb-937b-a54278f1ed13","Type":"ContainerStarted","Data":"f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb"}
Jul 08 17:08:39.351751 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.351723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" event={"ID":"2c2ba3ac-69f2-47cb-937b-a54278f1ed13","Type":"ContainerStarted","Data":"c8763f26dea695974157bf82463199aca50ce612990f0522e2470b2cf5de8c61"}
Jul 08 17:08:39.351927 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.351756    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:39.353411 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.353389    2575 generic.go:356] "Generic (PLEG): container finished" podID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerID="5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee" exitCode=2
Jul 08 17:08:39.353505 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.353437    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 08 17:08:39.353505 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.353447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" event={"ID":"4e911cc7-1bab-45a3-95a8-d6b37031c1c7","Type":"ContainerDied","Data":"5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee"}
Jul 08 17:08:39.370387 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.370349    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podStartSLOduration=1.370336434 podStartE2EDuration="1.370336434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:08:39.369011251 +0000 UTC m=+2060.358568010" watchObservedRunningTime="2026-07-08 17:08:39.370336434 +0000 UTC m=+2060.359893176"
Jul 08 17:08:39.375252 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.375234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:39.381677 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.378094    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls\") pod \"success-200-isvc-ab489-predictor-5448cc6d6f-rl89h\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") " pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:39.526877 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.526793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:39.660567 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:39.660539    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"]
Jul 08 17:08:39.661655 ip-10-0-135-143 kubenswrapper[2575]: W0708 17:08:39.661619    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dc77a7_b03c_4dad_a43b_96d7981d9400.slice/crio-7356be14878a86fa281f5f2c1b89540eb0c5b09f87fa3d67e878361a19663b17 WatchSource:0}: Error finding container 7356be14878a86fa281f5f2c1b89540eb0c5b09f87fa3d67e878361a19663b17: Status 404 returned error can't find the container with id 7356be14878a86fa281f5f2c1b89540eb0c5b09f87fa3d67e878361a19663b17
Jul 08 17:08:40.025708 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.025667    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 08 17:08:40.359342 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.359298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" event={"ID":"57dc77a7-b03c-4dad-a43b-96d7981d9400","Type":"ContainerStarted","Data":"bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965"}
Jul 08 17:08:40.359530 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.359366    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" event={"ID":"57dc77a7-b03c-4dad-a43b-96d7981d9400","Type":"ContainerStarted","Data":"6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d"}
Jul 08 17:08:40.359530 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.359400    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" event={"ID":"57dc77a7-b03c-4dad-a43b-96d7981d9400","Type":"ContainerStarted","Data":"7356be14878a86fa281f5f2c1b89540eb0c5b09f87fa3d67e878361a19663b17"}
Jul 08 17:08:40.359671 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.359577    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 08 17:08:40.359671 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.359621    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:40.359779 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.359699    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:40.360804 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.360780    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 08 17:08:40.376610 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:40.376561    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podStartSLOduration=2.37654485 podStartE2EDuration="2.37654485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:08:40.375460871 +0000 UTC m=+2061.365017634" watchObservedRunningTime="2026-07-08 17:08:40.37654485 +0000 UTC m=+2061.366101605"
Jul 08 17:08:41.030392 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.030299    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 08 17:08:41.364640 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.364599    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 08 17:08:41.781547 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.781523    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:08:41.797379 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.797346    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztzcl\" (UniqueName: \"kubernetes.io/projected/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-api-access-ztzcl\") pod \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") "
Jul 08 17:08:41.797544 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.797391    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-proxy-tls\") pod \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") "
Jul 08 17:08:41.797544 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.797419    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-rbac-proxy-sar-config\") pod \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\" (UID: \"4e911cc7-1bab-45a3-95a8-d6b37031c1c7\") "
Jul 08 17:08:41.797821 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.797799    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e911cc7-1bab-45a3-95a8-d6b37031c1c7" (UID: "4e911cc7-1bab-45a3-95a8-d6b37031c1c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:08:41.799818 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.799795    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e911cc7-1bab-45a3-95a8-d6b37031c1c7" (UID: "4e911cc7-1bab-45a3-95a8-d6b37031c1c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:08:41.800011 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.799990    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-api-access-ztzcl" (OuterVolumeSpecName: "kube-api-access-ztzcl") pod "4e911cc7-1bab-45a3-95a8-d6b37031c1c7" (UID: "4e911cc7-1bab-45a3-95a8-d6b37031c1c7"). InnerVolumeSpecName "kube-api-access-ztzcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:08:41.898957 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.898877    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztzcl\" (UniqueName: \"kubernetes.io/projected/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-api-access-ztzcl\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:41.898957 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.898903    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:41.898957 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:41.898914    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e911cc7-1bab-45a3-95a8-d6b37031c1c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:42.240556 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.240531    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:08:42.302525 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.302495    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-rbac-proxy-sar-config\") pod \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") "
Jul 08 17:08:42.302752 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.302555    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njmsh\" (UniqueName: \"kubernetes.io/projected/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-api-access-njmsh\") pod \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") "
Jul 08 17:08:42.302752 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.302627    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-proxy-tls\") pod \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\" (UID: \"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705\") "
Jul 08 17:08:42.302862 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.302843    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" (UID: "c2204dcf-bfa9-4818-8fe4-ba80bb0ab705"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:08:42.302926 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.302912    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:42.304609 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.304582    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" (UID: "c2204dcf-bfa9-4818-8fe4-ba80bb0ab705"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:08:42.304737 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.304618    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-api-access-njmsh" (OuterVolumeSpecName: "kube-api-access-njmsh") pod "c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" (UID: "c2204dcf-bfa9-4818-8fe4-ba80bb0ab705"). InnerVolumeSpecName "kube-api-access-njmsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:08:42.375552 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.375380    2575 generic.go:356] "Generic (PLEG): container finished" podID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerID="4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9" exitCode=0
Jul 08 17:08:42.375745 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.375553    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" event={"ID":"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705","Type":"ContainerDied","Data":"4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9"}
Jul 08 17:08:42.375745 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.375575    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"
Jul 08 17:08:42.375745 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.375671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6" event={"ID":"c2204dcf-bfa9-4818-8fe4-ba80bb0ab705","Type":"ContainerDied","Data":"69843e471f8a782639c85d14980f19a485b864909ffd9b1ee66ebeb43fc88f57"}
Jul 08 17:08:42.375745 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.375712    2575 scope.go:117] "RemoveContainer" containerID="de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3"
Jul 08 17:08:42.378284 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.378120    2575 generic.go:356] "Generic (PLEG): container finished" podID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerID="2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805" exitCode=0
Jul 08 17:08:42.378284 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.378182    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" event={"ID":"4e911cc7-1bab-45a3-95a8-d6b37031c1c7","Type":"ContainerDied","Data":"2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805"}
Jul 08 17:08:42.378284 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.378204    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"
Jul 08 17:08:42.378284 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.378242    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d" event={"ID":"4e911cc7-1bab-45a3-95a8-d6b37031c1c7","Type":"ContainerDied","Data":"c714155fd71869be4cd96079973cc31779b6806c080efdbfcb4397b5bb54da48"}
Jul 08 17:08:42.388103 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.388083    2575 scope.go:117] "RemoveContainer" containerID="4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9"
Jul 08 17:08:42.395564 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.395548    2575 scope.go:117] "RemoveContainer" containerID="de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3"
Jul 08 17:08:42.395822 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:42.395803    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3\": container with ID starting with de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3 not found: ID does not exist" containerID="de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3"
Jul 08 17:08:42.395958 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.395833    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3"} err="failed to get container status \"de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3\": rpc error: code = NotFound desc = could not find container \"de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3\": container with ID starting with de6408100d95e74e3e64509df7637acaea661773d959aa390385606dfc964ce3 not found: ID does not exist"
Jul 08 17:08:42.395958 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.395870    2575 scope.go:117] "RemoveContainer" containerID="4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9"
Jul 08 17:08:42.396160 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:42.396144    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9\": container with ID starting with 4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9 not found: ID does not exist" containerID="4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9"
Jul 08 17:08:42.396200 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.396167    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9"} err="failed to get container status \"4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9\": rpc error: code = NotFound desc = could not find container \"4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9\": container with ID starting with 4fdb36a6ed8a7c93735bb87c08bdc2295d55a4a4a7ffc85c9c7a9ba0827743f9 not found: ID does not exist"
Jul 08 17:08:42.396200 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.396182    2575 scope.go:117] "RemoveContainer" containerID="5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee"
Jul 08 17:08:42.401193 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.401172    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"]
Jul 08 17:08:42.403630 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.403611    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njmsh\" (UniqueName: \"kubernetes.io/projected/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-kube-api-access-njmsh\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:42.403720 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.403634    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:08:42.404347 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.404331    2575 scope.go:117] "RemoveContainer" containerID="2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805"
Jul 08 17:08:42.407629 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.407609    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2c597-predictor-6bbc7ccbcc-m66q6"]
Jul 08 17:08:42.411775 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.411757    2575 scope.go:117] "RemoveContainer" containerID="5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee"
Jul 08 17:08:42.412023 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:42.412003    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee\": container with ID starting with 5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee not found: ID does not exist" containerID="5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee"
Jul 08 17:08:42.412098 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.412033    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee"} err="failed to get container status \"5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee\": rpc error: code = NotFound desc = could not find container \"5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee\": container with ID starting with 5e88f04f5735cac70378b2eaeaf3901d1e067d6bf9fb3d09d9d58d7fc63c9bee not found: ID does not exist"
Jul 08 17:08:42.412098 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.412056    2575 scope.go:117] "RemoveContainer" containerID="2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805"
Jul 08 17:08:42.412311 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:08:42.412296    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805\": container with ID starting with 2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805 not found: ID does not exist" containerID="2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805"
Jul 08 17:08:42.412347 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.412315    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805"} err="failed to get container status \"2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805\": rpc error: code = NotFound desc = could not find container \"2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805\": container with ID starting with 2335acab36aa0f79d20fcd88cae6905822714a721311be64d4ec90263f768805 not found: ID does not exist"
Jul 08 17:08:42.420767 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.420717    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"]
Jul 08 17:08:42.429543 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:42.429523    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2c597-predictor-6bdcb48c76-7578d"]
Jul 08 17:08:43.608370 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:43.608338    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" path="/var/lib/kubelet/pods/4e911cc7-1bab-45a3-95a8-d6b37031c1c7/volumes"
Jul 08 17:08:43.608779 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:43.608764    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" path="/var/lib/kubelet/pods/c2204dcf-bfa9-4818-8fe4-ba80bb0ab705/volumes"
Jul 08 17:08:45.308709 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:45.308667    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 08 17:08:45.309137 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:45.308987    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 08 17:08:45.364666 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:45.364632    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:08:45.365068 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:45.365046    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 08 17:08:46.368966 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:46.368937    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:08:46.369540 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:46.369515    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 08 17:08:55.308663 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:55.308615    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 08 17:08:55.309117 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:55.308994    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 08 17:08:55.365991 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:55.365958    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 08 17:08:56.370308 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:08:56.370271    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 08 17:09:05.309044 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:05.309003    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 08 17:09:05.309610 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:05.309003    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 08 17:09:05.365668 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:05.365619    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 08 17:09:06.370078 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:06.370040    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 08 17:09:15.309612 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:15.309581    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:09:15.310085 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:15.309791    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:09:15.365199 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:15.365157    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 08 17:09:16.370355 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:16.370319    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 08 17:09:19.585288 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:19.585257    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:09:19.589282 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:19.589175    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:09:25.366125 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:25.366086    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 08 17:09:26.370443 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:26.370405    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 08 17:09:28.101476 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:28.101446    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:28.512565 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:28.512488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:28.611366 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:28.611333    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:35.366398 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:35.366371    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:09:36.370707 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:36.370674    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:09:48.693202 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:48.693170    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:49.020934 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:09:49.020859    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:11:11.623406 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:11:11.623368    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:14:19.617089 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:14:19.617055    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:14:19.620103 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:14:19.620084    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:16:08.399800 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:16:08.399771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:09.563970 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:16:09.563938    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:43.017548 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:16:43.017515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:18:03.299264 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.299180    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:18:03.399943 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.399912    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"]
Jul 08 17:18:03.400196 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.400168    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container" containerID="cri-o://6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d" gracePeriod=30
Jul 08 17:18:03.400302 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.400189    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kube-rbac-proxy" containerID="cri-o://bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965" gracePeriod=30
Jul 08 17:18:03.421230 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.421207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:18:03.442826 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.442799    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"]
Jul 08 17:18:03.443051 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.443022    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" containerID="cri-o://f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb" gracePeriod=30
Jul 08 17:18:03.443145 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.443053    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kube-rbac-proxy" containerID="cri-o://69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f" gracePeriod=30
Jul 08 17:18:03.507037 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.507010    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:18:03.760202 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.760168    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerID="69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f" exitCode=2
Jul 08 17:18:03.760366 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.760236    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" event={"ID":"2c2ba3ac-69f2-47cb-937b-a54278f1ed13","Type":"ContainerDied","Data":"69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f"}
Jul 08 17:18:03.761876 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.761857    2575 generic.go:356] "Generic (PLEG): container finished" podID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerID="bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965" exitCode=2
Jul 08 17:18:03.761976 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:03.761933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" event={"ID":"57dc77a7-b03c-4dad-a43b-96d7981d9400","Type":"ContainerDied","Data":"bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965"}
Jul 08 17:18:05.360137 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:05.360097    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 08 17:18:05.365427 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:05.365401    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 08 17:18:06.214772 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.214747    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:18:06.217523 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.217503    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:18:06.308113 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.308048    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r9nx\" (UniqueName: \"kubernetes.io/projected/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-api-access-5r9nx\") pod \"57dc77a7-b03c-4dad-a43b-96d7981d9400\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") "
Jul 08 17:18:06.308113 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.308090    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4g42\" (UniqueName: \"kubernetes.io/projected/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-api-access-l4g42\") pod \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") "
Jul 08 17:18:06.308343 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.308120    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls\") pod \"57dc77a7-b03c-4dad-a43b-96d7981d9400\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") "
Jul 08 17:18:06.308343 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.308143    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-proxy-tls\") pod \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") "
Jul 08 17:18:06.308343 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.308160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-rbac-proxy-sar-config\") pod \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\" (UID: \"2c2ba3ac-69f2-47cb-937b-a54278f1ed13\") "
Jul 08 17:18:06.308343 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.308183    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-rbac-proxy-sar-config\") pod \"57dc77a7-b03c-4dad-a43b-96d7981d9400\" (UID: \"57dc77a7-b03c-4dad-a43b-96d7981d9400\") "
Jul 08 17:18:06.308573 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.308547    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c2ba3ac-69f2-47cb-937b-a54278f1ed13" (UID: "2c2ba3ac-69f2-47cb-937b-a54278f1ed13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:18:06.308669 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.308617    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57dc77a7-b03c-4dad-a43b-96d7981d9400" (UID: "57dc77a7-b03c-4dad-a43b-96d7981d9400"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:18:06.310247 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.310214    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c2ba3ac-69f2-47cb-937b-a54278f1ed13" (UID: "2c2ba3ac-69f2-47cb-937b-a54278f1ed13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:18:06.310344 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.310253    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-api-access-l4g42" (OuterVolumeSpecName: "kube-api-access-l4g42") pod "2c2ba3ac-69f2-47cb-937b-a54278f1ed13" (UID: "2c2ba3ac-69f2-47cb-937b-a54278f1ed13"). InnerVolumeSpecName "kube-api-access-l4g42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:18:06.310344 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.310274    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-api-access-5r9nx" (OuterVolumeSpecName: "kube-api-access-5r9nx") pod "57dc77a7-b03c-4dad-a43b-96d7981d9400" (UID: "57dc77a7-b03c-4dad-a43b-96d7981d9400"). InnerVolumeSpecName "kube-api-access-5r9nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:18:06.310344 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.310272    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57dc77a7-b03c-4dad-a43b-96d7981d9400" (UID: "57dc77a7-b03c-4dad-a43b-96d7981d9400"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:18:06.409344 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.409322    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:18:06.409344 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.409344    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:18:06.409685 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.409354    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r9nx\" (UniqueName: \"kubernetes.io/projected/57dc77a7-b03c-4dad-a43b-96d7981d9400-kube-api-access-5r9nx\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:18:06.409685 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.409362    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4g42\" (UniqueName: \"kubernetes.io/projected/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-kube-api-access-l4g42\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:18:06.409685 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.409371    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57dc77a7-b03c-4dad-a43b-96d7981d9400-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:18:06.409685 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.409380    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2ba3ac-69f2-47cb-937b-a54278f1ed13-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:18:06.777401 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.777375    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerID="f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb" exitCode=0
Jul 08 17:18:06.777578 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.777434    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"
Jul 08 17:18:06.777578 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.777456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" event={"ID":"2c2ba3ac-69f2-47cb-937b-a54278f1ed13","Type":"ContainerDied","Data":"f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb"}
Jul 08 17:18:06.777578 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.777516    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9" event={"ID":"2c2ba3ac-69f2-47cb-937b-a54278f1ed13","Type":"ContainerDied","Data":"c8763f26dea695974157bf82463199aca50ce612990f0522e2470b2cf5de8c61"}
Jul 08 17:18:06.777578 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.777552    2575 scope.go:117] "RemoveContainer" containerID="69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f"
Jul 08 17:18:06.779071 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.779051    2575 generic.go:356] "Generic (PLEG): container finished" podID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerID="6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d" exitCode=0
Jul 08 17:18:06.779192 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.779099    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"
Jul 08 17:18:06.779192 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.779100    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" event={"ID":"57dc77a7-b03c-4dad-a43b-96d7981d9400","Type":"ContainerDied","Data":"6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d"}
Jul 08 17:18:06.779306 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.779232    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h" event={"ID":"57dc77a7-b03c-4dad-a43b-96d7981d9400","Type":"ContainerDied","Data":"7356be14878a86fa281f5f2c1b89540eb0c5b09f87fa3d67e878361a19663b17"}
Jul 08 17:18:06.787056 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.787039    2575 scope.go:117] "RemoveContainer" containerID="f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb"
Jul 08 17:18:06.794179 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.794159    2575 scope.go:117] "RemoveContainer" containerID="69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f"
Jul 08 17:18:06.794419 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:18:06.794402    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f\": container with ID starting with 69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f not found: ID does not exist" containerID="69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f"
Jul 08 17:18:06.794466 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.794427    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f"} err="failed to get container status \"69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f\": rpc error: code = NotFound desc = could not find container \"69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f\": container with ID starting with 69081659749fb1b7d2153efefe9f49bb512cc28b9210f7c062fa349af25e7b5f not found: ID does not exist"
Jul 08 17:18:06.794466 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.794443    2575 scope.go:117] "RemoveContainer" containerID="f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb"
Jul 08 17:18:06.794756 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:18:06.794741    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb\": container with ID starting with f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb not found: ID does not exist" containerID="f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb"
Jul 08 17:18:06.794817 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.794759    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb"} err="failed to get container status \"f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb\": rpc error: code = NotFound desc = could not find container \"f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb\": container with ID starting with f409dd60d3be19ac3524522a4983da342637bcc42bc558d81d4105582f2558fb not found: ID does not exist"
Jul 08 17:18:06.794817 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.794772    2575 scope.go:117] "RemoveContainer" containerID="bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965"
Jul 08 17:18:06.804453 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.804436    2575 scope.go:117] "RemoveContainer" containerID="6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d"
Jul 08 17:18:06.811492 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.811414    2575 scope.go:117] "RemoveContainer" containerID="bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965"
Jul 08 17:18:06.811730 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:18:06.811711    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965\": container with ID starting with bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965 not found: ID does not exist" containerID="bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965"
Jul 08 17:18:06.811802 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.811734    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965"} err="failed to get container status \"bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965\": rpc error: code = NotFound desc = could not find container \"bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965\": container with ID starting with bfb7be6adca521e90e7a44a3218b4a0153e2a8319ee39c4d3f9759e83919a965 not found: ID does not exist"
Jul 08 17:18:06.811802 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.811748    2575 scope.go:117] "RemoveContainer" containerID="6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d"
Jul 08 17:18:06.812000 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:18:06.811983    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d\": container with ID starting with 6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d not found: ID does not exist" containerID="6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d"
Jul 08 17:18:06.812053 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.812005    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d"} err="failed to get container status \"6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d\": rpc error: code = NotFound desc = could not find container \"6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d\": container with ID starting with 6304fe1650291057f69ae65f23152293c513e6eef17c67d32a7d8674933baf3d not found: ID does not exist"
Jul 08 17:18:06.812974 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.812957    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"]
Jul 08 17:18:06.818367 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.818347    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab489-predictor-5448cc6d6f-rl89h"]
Jul 08 17:18:06.827840 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.827822    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"]
Jul 08 17:18:06.830608 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:06.830587    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab489-predictor-76897c7688-74fs9"]
Jul 08 17:18:07.608054 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:07.608024    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" path="/var/lib/kubelet/pods/2c2ba3ac-69f2-47cb-937b-a54278f1ed13/volumes"
Jul 08 17:18:07.608427 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:18:07.608413    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" path="/var/lib/kubelet/pods/57dc77a7-b03c-4dad-a43b-96d7981d9400/volumes"
Jul 08 17:19:19.640160 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:19:19.640134    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:19:19.644705 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:19:19.644687    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:20:19.289889 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:20:19.289855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:24:19.664211 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:24:19.664133    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:24:19.670561 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:24:19.670539    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:25:47.499210 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:47.499173    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:25:47.613703 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:47.613675    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"]
Jul 08 17:25:47.664028 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:47.664001    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"]
Jul 08 17:25:47.664248 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:47.664201    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container" containerID="cri-o://8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8" gracePeriod=30
Jul 08 17:25:47.664316 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:47.664248    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kube-rbac-proxy" containerID="cri-o://11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff" gracePeriod=30
Jul 08 17:25:47.692180 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:47.692155    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-serving-cert: secret "success-200-isvc-d09b9-predictor-serving-cert" not found
Jul 08 17:25:47.692288 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:47.692219    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls podName:7f7c6494-5c28-4353-8f42-620c8e8d277b nodeName:}" failed. No retries permitted until 2026-07-08 17:25:48.192203642 +0000 UTC m=+3089.181760363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls") pod "success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" (UID: "7f7c6494-5c28-4353-8f42-620c8e8d277b") : secret "success-200-isvc-d09b9-predictor-serving-cert" not found
Jul 08 17:25:48.196780 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:48.196754    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-serving-cert: secret "success-200-isvc-d09b9-predictor-serving-cert" not found
Jul 08 17:25:48.196934 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:48.196827    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls podName:7f7c6494-5c28-4353-8f42-620c8e8d277b nodeName:}" failed. No retries permitted until 2026-07-08 17:25:49.196809953 +0000 UTC m=+3090.186366694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls") pod "success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" (UID: "7f7c6494-5c28-4353-8f42-620c8e8d277b") : secret "success-200-isvc-d09b9-predictor-serving-cert" not found
Jul 08 17:25:48.554518 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:48.554434    2575 generic.go:356] "Generic (PLEG): container finished" podID="86378337-6f02-4b24-b099-66c502100f55" containerID="11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff" exitCode=2
Jul 08 17:25:48.554926 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:48.554516    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" event={"ID":"86378337-6f02-4b24-b099-66c502100f55","Type":"ContainerDied","Data":"11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff"}
Jul 08 17:25:48.554926 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:48.554596    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container" containerID="cri-o://58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7" gracePeriod=30
Jul 08 17:25:48.554926 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:48.554612    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kube-rbac-proxy" containerID="cri-o://302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a" gracePeriod=30
Jul 08 17:25:48.849037 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:48.849012    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:25:48.980670 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:48.980618    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:25:49.081776 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:49.081744    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:25:49.205113 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:49.205030    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-serving-cert: secret "success-200-isvc-d09b9-predictor-serving-cert" not found
Jul 08 17:25:49.205113 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:49.205111    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls podName:7f7c6494-5c28-4353-8f42-620c8e8d277b nodeName:}" failed. No retries permitted until 2026-07-08 17:25:51.205090224 +0000 UTC m=+3092.194646949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls") pod "success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" (UID: "7f7c6494-5c28-4353-8f42-620c8e8d277b") : secret "success-200-isvc-d09b9-predictor-serving-cert" not found
Jul 08 17:25:49.560134 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:49.560051    2575 generic.go:356] "Generic (PLEG): container finished" podID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerID="302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a" exitCode=2
Jul 08 17:25:49.560134 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:49.560094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" event={"ID":"7f7c6494-5c28-4353-8f42-620c8e8d277b","Type":"ContainerDied","Data":"302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a"}
Jul 08 17:25:50.303012 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.302975    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 08 17:25:50.303012 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.302991    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 08 17:25:50.418722 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.418700    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:25:50.516184 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.516098    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86378337-6f02-4b24-b099-66c502100f55-proxy-tls\") pod \"86378337-6f02-4b24-b099-66c502100f55\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") "
Jul 08 17:25:50.516184 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.516167    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jthdb\" (UniqueName: \"kubernetes.io/projected/86378337-6f02-4b24-b099-66c502100f55-kube-api-access-jthdb\") pod \"86378337-6f02-4b24-b099-66c502100f55\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") "
Jul 08 17:25:50.516405 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.516220    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86378337-6f02-4b24-b099-66c502100f55-kube-rbac-proxy-sar-config\") pod \"86378337-6f02-4b24-b099-66c502100f55\" (UID: \"86378337-6f02-4b24-b099-66c502100f55\") "
Jul 08 17:25:50.516593 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.516569    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86378337-6f02-4b24-b099-66c502100f55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86378337-6f02-4b24-b099-66c502100f55" (UID: "86378337-6f02-4b24-b099-66c502100f55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:25:50.518131 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.518110    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86378337-6f02-4b24-b099-66c502100f55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86378337-6f02-4b24-b099-66c502100f55" (UID: "86378337-6f02-4b24-b099-66c502100f55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:25:50.518227 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.518152    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86378337-6f02-4b24-b099-66c502100f55-kube-api-access-jthdb" (OuterVolumeSpecName: "kube-api-access-jthdb") pod "86378337-6f02-4b24-b099-66c502100f55" (UID: "86378337-6f02-4b24-b099-66c502100f55"). InnerVolumeSpecName "kube-api-access-jthdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:25:50.565522 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.565493    2575 generic.go:356] "Generic (PLEG): container finished" podID="86378337-6f02-4b24-b099-66c502100f55" containerID="8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8" exitCode=0
Jul 08 17:25:50.565926 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.565547    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"
Jul 08 17:25:50.565926 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.565564    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" event={"ID":"86378337-6f02-4b24-b099-66c502100f55","Type":"ContainerDied","Data":"8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8"}
Jul 08 17:25:50.565926 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.565670    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc" event={"ID":"86378337-6f02-4b24-b099-66c502100f55","Type":"ContainerDied","Data":"30335bdfe00de4193d3a196f5bb0d1ae46a48b33889017e649ec4afbf8525927"}
Jul 08 17:25:50.565926 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.565701    2575 scope.go:117] "RemoveContainer" containerID="11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff"
Jul 08 17:25:50.574267 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.574252    2575 scope.go:117] "RemoveContainer" containerID="8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8"
Jul 08 17:25:50.581357 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.581336    2575 scope.go:117] "RemoveContainer" containerID="11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff"
Jul 08 17:25:50.581587 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:50.581568    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff\": container with ID starting with 11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff not found: ID does not exist" containerID="11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff"
Jul 08 17:25:50.581683 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.581593    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff"} err="failed to get container status \"11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff\": rpc error: code = NotFound desc = could not find container \"11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff\": container with ID starting with 11e3b7c6de3a0cc0b3040221d64a0ad683cbc0477393942e58d36a57f3d6cdff not found: ID does not exist"
Jul 08 17:25:50.581683 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.581610    2575 scope.go:117] "RemoveContainer" containerID="8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8"
Jul 08 17:25:50.581873 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:50.581854    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8\": container with ID starting with 8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8 not found: ID does not exist" containerID="8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8"
Jul 08 17:25:50.581933 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.581878    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8"} err="failed to get container status \"8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8\": rpc error: code = NotFound desc = could not find container \"8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8\": container with ID starting with 8bc5f305a1e7548af01d1a3e22337057fd826176e4d01bcc1e6f1be7b6f5adf8 not found: ID does not exist"
Jul 08 17:25:50.587010 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.586896    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"]
Jul 08 17:25:50.588471 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.588452    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d09b9-predictor-6cb94d46cb-489qc"]
Jul 08 17:25:50.617768 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.617748    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86378337-6f02-4b24-b099-66c502100f55-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:25:50.617768 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.617767    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jthdb\" (UniqueName: \"kubernetes.io/projected/86378337-6f02-4b24-b099-66c502100f55-kube-api-access-jthdb\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:25:50.617865 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:50.617777    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86378337-6f02-4b24-b099-66c502100f55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:25:51.222400 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:51.222375    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-serving-cert: secret "success-200-isvc-d09b9-predictor-serving-cert" not found
Jul 08 17:25:51.222518 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:51.222435    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls podName:7f7c6494-5c28-4353-8f42-620c8e8d277b nodeName:}" failed. No retries permitted until 2026-07-08 17:25:55.222422444 +0000 UTC m=+3096.211979165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls") pod "success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" (UID: "7f7c6494-5c28-4353-8f42-620c8e8d277b") : secret "success-200-isvc-d09b9-predictor-serving-cert" not found
Jul 08 17:25:51.402142 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.402121    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:25:51.525106 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.525041    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls\") pod \"7f7c6494-5c28-4353-8f42-620c8e8d277b\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") "
Jul 08 17:25:51.525223 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.525119    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwm8q\" (UniqueName: \"kubernetes.io/projected/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-api-access-wwm8q\") pod \"7f7c6494-5c28-4353-8f42-620c8e8d277b\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") "
Jul 08 17:25:51.525282 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.525266    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-rbac-proxy-sar-config\") pod \"7f7c6494-5c28-4353-8f42-620c8e8d277b\" (UID: \"7f7c6494-5c28-4353-8f42-620c8e8d277b\") "
Jul 08 17:25:51.525567 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.525533    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f7c6494-5c28-4353-8f42-620c8e8d277b" (UID: "7f7c6494-5c28-4353-8f42-620c8e8d277b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:25:51.527071 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.527051    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-api-access-wwm8q" (OuterVolumeSpecName: "kube-api-access-wwm8q") pod "7f7c6494-5c28-4353-8f42-620c8e8d277b" (UID: "7f7c6494-5c28-4353-8f42-620c8e8d277b"). InnerVolumeSpecName "kube-api-access-wwm8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:25:51.527167 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.527078    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f7c6494-5c28-4353-8f42-620c8e8d277b" (UID: "7f7c6494-5c28-4353-8f42-620c8e8d277b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:25:51.571090 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.571062    2575 generic.go:356] "Generic (PLEG): container finished" podID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerID="58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7" exitCode=0
Jul 08 17:25:51.571402 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.571104    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" event={"ID":"7f7c6494-5c28-4353-8f42-620c8e8d277b","Type":"ContainerDied","Data":"58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7"}
Jul 08 17:25:51.571402 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.571109    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"
Jul 08 17:25:51.571402 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.571150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb" event={"ID":"7f7c6494-5c28-4353-8f42-620c8e8d277b","Type":"ContainerDied","Data":"c0e477dfb5d12e63321e0a3d6e46e0029c6bb7d0e93aaf45c71690539e02a147"}
Jul 08 17:25:51.571402 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.571175    2575 scope.go:117] "RemoveContainer" containerID="302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a"
Jul 08 17:25:51.579510 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.579492    2575 scope.go:117] "RemoveContainer" containerID="58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7"
Jul 08 17:25:51.586351 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.586330    2575 scope.go:117] "RemoveContainer" containerID="302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a"
Jul 08 17:25:51.586611 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:51.586591    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a\": container with ID starting with 302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a not found: ID does not exist" containerID="302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a"
Jul 08 17:25:51.586670 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.586619    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a"} err="failed to get container status \"302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a\": rpc error: code = NotFound desc = could not find container \"302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a\": container with ID starting with 302b3a4be1e7f247d9198d0a545a524b52a918767a96d34b969967e2fba33a6a not found: ID does not exist"
Jul 08 17:25:51.586670 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.586636    2575 scope.go:117] "RemoveContainer" containerID="58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7"
Jul 08 17:25:51.586877 ip-10-0-135-143 kubenswrapper[2575]: E0708 17:25:51.586861    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7\": container with ID starting with 58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7 not found: ID does not exist" containerID="58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7"
Jul 08 17:25:51.586917 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.586883    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7"} err="failed to get container status \"58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7\": rpc error: code = NotFound desc = could not find container \"58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7\": container with ID starting with 58eca329e0066e76ab144e366b0cc87544006a8dd4ff8e2f6084ce09501634d7 not found: ID does not exist"
Jul 08 17:25:51.591854 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.591833    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"]
Jul 08 17:25:51.594864 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.594844    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d09b9-predictor-75cd6867df-q6mgb"]
Jul 08 17:25:51.607827 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.607808    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" path="/var/lib/kubelet/pods/7f7c6494-5c28-4353-8f42-620c8e8d277b/volumes"
Jul 08 17:25:51.608212 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.608200    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86378337-6f02-4b24-b099-66c502100f55" path="/var/lib/kubelet/pods/86378337-6f02-4b24-b099-66c502100f55/volumes"
Jul 08 17:25:51.626759 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.626741    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwm8q\" (UniqueName: \"kubernetes.io/projected/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-api-access-wwm8q\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:25:51.626759 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.626759    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7c6494-5c28-4353-8f42-620c8e8d277b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:25:51.626873 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:51.626770    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7c6494-5c28-4353-8f42-620c8e8d277b-proxy-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 08 17:25:56.564685 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:25:56.564640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:17.314031 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:17.313998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:20.259408 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.259377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:20.405106 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405076    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zkvj/must-gather-r669v"]
Jul 08 17:26:20.405372 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405357    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405372 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405370    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405381    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405386    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405393    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405400    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405411    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405417    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405423    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405429    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405436    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405441    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405447    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405452 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405452    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405458    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405463    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405469    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405474    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405480    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405485    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405491    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405496    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405502    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405506    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405559    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405567    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e911cc7-1bab-45a3-95a8-d6b37031c1c7" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405573    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405579    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405584    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7c6494-5c28-4353-8f42-620c8e8d277b" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405589    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="86378337-6f02-4b24-b099-66c502100f55" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405594    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405600    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405604    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405610    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2204dcf-bfa9-4818-8fe4-ba80bb0ab705" containerName="kserve-container"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405614    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2ba3ac-69f2-47cb-937b-a54278f1ed13" containerName="kube-rbac-proxy"
Jul 08 17:26:20.405885 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.405619    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="57dc77a7-b03c-4dad-a43b-96d7981d9400" containerName="kserve-container"
Jul 08 17:26:20.409373 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.409347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:20.409896 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.409881    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zkvj/must-gather-r669v"
Jul 08 17:26:20.412301 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.412280    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zkvj\"/\"openshift-service-ca.crt\""
Jul 08 17:26:20.412403 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.412299    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9zkvj\"/\"default-dockercfg-blrjz\""
Jul 08 17:26:20.413280 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.413264    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zkvj\"/\"kube-root-ca.crt\""
Jul 08 17:26:20.416943 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.416918    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zkvj/must-gather-r669v"]
Jul 08 17:26:20.477277 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.477245    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:20.531477 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.531417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cb6f\" (UniqueName: \"kubernetes.io/projected/4de406a5-5377-46c9-82dd-3c37969421e6-kube-api-access-9cb6f\") pod \"must-gather-r669v\" (UID: \"4de406a5-5377-46c9-82dd-3c37969421e6\") " pod="openshift-must-gather-9zkvj/must-gather-r669v"
Jul 08 17:26:20.531477 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.531459    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4de406a5-5377-46c9-82dd-3c37969421e6-must-gather-output\") pod \"must-gather-r669v\" (UID: \"4de406a5-5377-46c9-82dd-3c37969421e6\") " pod="openshift-must-gather-9zkvj/must-gather-r669v"
Jul 08 17:26:20.632251 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.632217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cb6f\" (UniqueName: \"kubernetes.io/projected/4de406a5-5377-46c9-82dd-3c37969421e6-kube-api-access-9cb6f\") pod \"must-gather-r669v\" (UID: \"4de406a5-5377-46c9-82dd-3c37969421e6\") " pod="openshift-must-gather-9zkvj/must-gather-r669v"
Jul 08 17:26:20.632390 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.632265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4de406a5-5377-46c9-82dd-3c37969421e6-must-gather-output\") pod \"must-gather-r669v\" (UID: \"4de406a5-5377-46c9-82dd-3c37969421e6\") " pod="openshift-must-gather-9zkvj/must-gather-r669v"
Jul 08 17:26:20.632589 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.632574    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4de406a5-5377-46c9-82dd-3c37969421e6-must-gather-output\") pod \"must-gather-r669v\" (UID: \"4de406a5-5377-46c9-82dd-3c37969421e6\") " pod="openshift-must-gather-9zkvj/must-gather-r669v"
Jul 08 17:26:20.639784 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.639758    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cb6f\" (UniqueName: \"kubernetes.io/projected/4de406a5-5377-46c9-82dd-3c37969421e6-kube-api-access-9cb6f\") pod \"must-gather-r669v\" (UID: \"4de406a5-5377-46c9-82dd-3c37969421e6\") " pod="openshift-must-gather-9zkvj/must-gather-r669v"
Jul 08 17:26:20.732909 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.732880    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zkvj/must-gather-r669v"
Jul 08 17:26:20.855716 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.855686    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zkvj/must-gather-r669v"]
Jul 08 17:26:20.858443 ip-10-0-135-143 kubenswrapper[2575]: W0708 17:26:20.858416    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4de406a5_5377_46c9_82dd_3c37969421e6.slice/crio-f3cd9448f6f36b3c46b8451f014453aebead4d457a77e14df31434e93c9353a0 WatchSource:0}: Error finding container f3cd9448f6f36b3c46b8451f014453aebead4d457a77e14df31434e93c9353a0: Status 404 returned error can't find the container with id f3cd9448f6f36b3c46b8451f014453aebead4d457a77e14df31434e93c9353a0
Jul 08 17:26:20.860362 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:20.860344    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 17:26:21.682372 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:21.682338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zkvj/must-gather-r669v" event={"ID":"4de406a5-5377-46c9-82dd-3c37969421e6","Type":"ContainerStarted","Data":"f3cd9448f6f36b3c46b8451f014453aebead4d457a77e14df31434e93c9353a0"}
Jul 08 17:26:22.688692 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:22.688638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zkvj/must-gather-r669v" event={"ID":"4de406a5-5377-46c9-82dd-3c37969421e6","Type":"ContainerStarted","Data":"2062e321456a2e459fce17f8b7541bcfca2971570a12ee5e7eca14965b03ebad"}
Jul 08 17:26:22.689187 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:22.688708    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zkvj/must-gather-r669v" event={"ID":"4de406a5-5377-46c9-82dd-3c37969421e6","Type":"ContainerStarted","Data":"8099dc23ea5d352aaee0df46933b5122f5fc3d7e38e014b2942daea250a52104"}
Jul 08 17:26:22.704929 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:22.704879    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9zkvj/must-gather-r669v" podStartSLOduration=1.420014382 podStartE2EDuration="2.704864459s" totalImagesPullingTime="1.284850077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:26:22.704756419 +0000 UTC m=+3123.694313163" watchObservedRunningTime="2026-07-08 17:26:22.704864459 +0000 UTC m=+3123.694421201"
Jul 08 17:26:23.630826 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:23.630798    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v8x8z_2a323b89-9740-4ab1-9d0d-1ca2bec3d93d/global-pull-secret-syncer/0.log"
Jul 08 17:26:23.724680 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:23.724632    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log"
Jul 08 17:26:23.863058 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:23.863029    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-143.ec2.internal_c73243dc8a0860bf821a5ab616d0670d/haproxy/0.log"
Jul 08 17:26:25.971507 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:25.971456    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:26.025752 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:26.025719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8snv_1a975ef5-c4da-4258-a07d-41d5259defde/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:28.077395 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:28.077310    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-gd5xv_afcc2e47-6a0a-498e-9e3d-3ab061a32e71/cluster-monitoring-operator/0.log"
Jul 08 17:26:28.608759 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:28.608729    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxgqt_81c52002-1b8f-4e5c-be4d-88f75d65dc00/node-exporter/0.log"
Jul 08 17:26:28.632699 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:28.632606    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxgqt_81c52002-1b8f-4e5c-be4d-88f75d65dc00/kube-rbac-proxy/0.log"
Jul 08 17:26:28.657106 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:28.657079    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxgqt_81c52002-1b8f-4e5c-be4d-88f75d65dc00/init-textfile/0.log"
Jul 08 17:26:31.485675 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.485636    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64d9866d66-z6bdl_0b192d5b-324e-4a8d-8863-bb48b3d4ed63/console/0.log"
Jul 08 17:26:31.523532 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.523505    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-4zfwc_f66ead7a-f82f-4786-a47c-51e175651730/download-server/0.log"
Jul 08 17:26:31.552602 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.552574    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"]
Jul 08 17:26:31.557282 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.557260    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.570713 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.570684    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"]
Jul 08 17:26:31.750430 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.750347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-proc\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.750430 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.750400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-lib-modules\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.750674 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.750450    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-sys\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.750674 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.750488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-podres\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.750674 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.750503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dzj\" (UniqueName: \"kubernetes.io/projected/d9534547-8d7b-41c3-a6db-fb054328d2dc-kube-api-access-27dzj\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852044 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-sys\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852323 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-podres\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852444 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852424    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-podres\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852510 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dzj\" (UniqueName: \"kubernetes.io/projected/d9534547-8d7b-41c3-a6db-fb054328d2dc-kube-api-access-27dzj\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852575 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852126    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-sys\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852631 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-proc\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852719 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852686    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-proc\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852719 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-lib-modules\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.852827 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.852816    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9534547-8d7b-41c3-a6db-fb054328d2dc-lib-modules\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.861297 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.861268    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dzj\" (UniqueName: \"kubernetes.io/projected/d9534547-8d7b-41c3-a6db-fb054328d2dc-kube-api-access-27dzj\") pod \"perf-node-gather-daemonset-5wfkf\" (UID: \"d9534547-8d7b-41c3-a6db-fb054328d2dc\") " pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:31.891208 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:31.891176    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:32.064240 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:32.064191    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"]
Jul 08 17:26:32.740091 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:32.740055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf" event={"ID":"d9534547-8d7b-41c3-a6db-fb054328d2dc","Type":"ContainerStarted","Data":"7fe81aede0e4d4a677dd718a639d3b2ecbfb02890cb3502a7a4a295080beef3e"}
Jul 08 17:26:32.740556 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:32.740111    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf" event={"ID":"d9534547-8d7b-41c3-a6db-fb054328d2dc","Type":"ContainerStarted","Data":"567c8f0aabaa45ed6650665ee25283502749289bef93fc2c6a96b55b00272ac4"}
Jul 08 17:26:32.740556 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:32.740135    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:32.756987 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:32.756944    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf" podStartSLOduration=1.756927291 podStartE2EDuration="1.756927291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:26:32.755960198 +0000 UTC m=+3133.745516953" watchObservedRunningTime="2026-07-08 17:26:32.756927291 +0000 UTC m=+3133.746484050"
Jul 08 17:26:33.052513 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:33.052440    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p8lx6_fa748963-12ef-4075-8741-ab58379b8a60/dns/0.log"
Jul 08 17:26:33.078038 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:33.078011    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p8lx6_fa748963-12ef-4075-8741-ab58379b8a60/kube-rbac-proxy/0.log"
Jul 08 17:26:33.260841 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:33.260816    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vtcrd_370616ca-420a-4679-91d6-9c035007c735/dns-node-resolver/0.log"
Jul 08 17:26:33.734788 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:33.734757    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-759b5bcd4-7fjbt_5801ea7f-3647-45a1-8a15-72a7a5b9b8c6/registry/0.log"
Jul 08 17:26:33.786140 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:33.786091    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8p82f_e60e3631-e481-4969-9b02-2db258f40abc/node-ca/0.log"
Jul 08 17:26:34.607942 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:34.607916    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f8778bf4f-tlgl8_9edb90cb-255d-4637-9bd9-14e8c87c3c06/router/0.log"
Jul 08 17:26:34.973780 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:34.973710    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-klkjg_f0dfd2e0-9a06-4c80-b124-7a84a7e92543/serve-healthcheck-canary/0.log"
Jul 08 17:26:35.384524 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:35.384491    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-mdgr8_650f88a0-c7d3-44d7-b457-d81d1584808c/insights-operator/0.log"
Jul 08 17:26:35.385639 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:35.385618    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-mdgr8_650f88a0-c7d3-44d7-b457-d81d1584808c/insights-operator/1.log"
Jul 08 17:26:35.558171 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:35.558149    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-psrtc_9099ba48-3525-45b0-a418-c0f0e9d4ba6f/kube-rbac-proxy/0.log"
Jul 08 17:26:35.580467 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:35.580447    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-psrtc_9099ba48-3525-45b0-a418-c0f0e9d4ba6f/exporter/0.log"
Jul 08 17:26:35.602879 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:35.602858    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-psrtc_9099ba48-3525-45b0-a418-c0f0e9d4ba6f/extractor/0.log"
Jul 08 17:26:37.649808 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:37.649778    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-fntrg_0b46c0ef-bf63-4cf7-a413-11e6351a4def/server/0.log"
Jul 08 17:26:37.764997 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:37.764962    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-xnzct_d93a62a9-b35c-47aa-a075-d85d06f31068/manager/0.log"
Jul 08 17:26:37.849363 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:37.849323    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-rnhkw_e86c348f-0af6-40dc-8789-cee488757371/seaweedfs/0.log"
Jul 08 17:26:38.753813 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:38.753778    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9zkvj/perf-node-gather-daemonset-5wfkf"
Jul 08 17:26:42.175088 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:42.175045    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-hgjls_0a874a90-715a-43b8-9e06-99ade98392b5/migrator/0.log"
Jul 08 17:26:42.197785 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:42.197757    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-hgjls_0a874a90-715a-43b8-9e06-99ade98392b5/graceful-termination/0.log"
Jul 08 17:26:43.526775 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:43.526738    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6bqfr_81e545bb-0f25-4af4-b5fe-6ef6ba0edcd6/kube-multus/0.log"
Jul 08 17:26:43.758204 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:43.758173    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bx962_2840b062-0201-4147-bc5e-17ceb2652dc7/kube-multus-additional-cni-plugins/0.log"
Jul 08 17:26:43.781199 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:43.781119    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bx962_2840b062-0201-4147-bc5e-17ceb2652dc7/egress-router-binary-copy/0.log"
Jul 08 17:26:43.803386 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:43.803361    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bx962_2840b062-0201-4147-bc5e-17ceb2652dc7/cni-plugins/0.log"
Jul 08 17:26:43.827123 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:43.827103    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bx962_2840b062-0201-4147-bc5e-17ceb2652dc7/bond-cni-plugin/0.log"
Jul 08 17:26:43.849001 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:43.848977    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bx962_2840b062-0201-4147-bc5e-17ceb2652dc7/routeoverride-cni/0.log"
Jul 08 17:26:43.872404 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:43.872385    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bx962_2840b062-0201-4147-bc5e-17ceb2652dc7/whereabouts-cni-bincopy/0.log"
Jul 08 17:26:43.895862 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:43.895838    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bx962_2840b062-0201-4147-bc5e-17ceb2652dc7/whereabouts-cni/0.log"
Jul 08 17:26:44.141859 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:44.141839    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4x4wh_8751bafd-f26c-47bd-8fa8-f24755c1fb18/network-metrics-daemon/0.log"
Jul 08 17:26:44.171851 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:44.171835    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4x4wh_8751bafd-f26c-47bd-8fa8-f24755c1fb18/kube-rbac-proxy/0.log"
Jul 08 17:26:45.330613 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.330585    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-controller/0.log"
Jul 08 17:26:45.347207 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.347187    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/0.log"
Jul 08 17:26:45.347324 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.347306    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovn-acl-logging/1.log"
Jul 08 17:26:45.365946 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.365922    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/kube-rbac-proxy-node/0.log"
Jul 08 17:26:45.391258 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.391236    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/kube-rbac-proxy-ovn-metrics/0.log"
Jul 08 17:26:45.413358 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.413338    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/northd/0.log"
Jul 08 17:26:45.442169 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.442149    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/nbdb/0.log"
Jul 08 17:26:45.467767 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.467747    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/sbdb/0.log"
Jul 08 17:26:45.518053 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:45.518035    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pcqm_12e4acd4-8b5f-40ae-b2cd-2399a39d43d8/ovnkube-controller/0.log"
Jul 08 17:26:46.804428 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:46.804400    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dtcdw_2f1fa945-7def-4e82-9da9-78010a748562/network-check-target-container/0.log"
Jul 08 17:26:47.898691 ip-10-0-135-143 kubenswrapper[2575]: I0708 17:26:47.898599    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gjvrt_dfb0ebd2-1b79-45cb-893f-edabc9497184/iptables-alerter/0.log"