Jul 13 18:50:51.973310 ip-10-0-139-100 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.516555    2587 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520550    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520564    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520567    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520570    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520574    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520576    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520579    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:54.272372 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520582    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:54.266999 ip-10-0-139-100 systemd[1]: Started Kubernetes Kubelet.
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520584    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520587    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520590    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520592    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520597    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520599    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520608    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520611    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520614    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520617    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520619    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520622    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520626    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520630    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520633    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520636    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520639    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520642    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:54.274118 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520645    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520647    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520650    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520653    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520655    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520658    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520660    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520663    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520665    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520668    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520670    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520673    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520675    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520677    2587 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520680    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520684    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520688    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520691    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520695    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520698    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:54.274989 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520701    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520704    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520706    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520709    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520711    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520714    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520717    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520719    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520723    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520725    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520728    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520731    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520733    2587 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520736    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520738    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520741    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520743    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520747    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520750    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:54.275944 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520753    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520756    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520759    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520761    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520764    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520766    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520770    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520772    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520775    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520778    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520781    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520784    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520786    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520788    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520792    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520795    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520798    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520801    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520804    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:54.276532 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520806    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520809    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520811    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520814    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520816    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520818    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520821    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520823    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520826    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520829    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520831    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520834    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520836    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520839    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.520841    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521570    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521577    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521580    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521583    2587 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521585    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:54.277283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521589    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521592    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521595    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521598    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521601    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521603    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521606    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521609    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521611    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521614    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521617    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521620    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521622    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521625    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521628    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521630    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521633    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521635    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521638    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521640    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:54.277885 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521643    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521645    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521648    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521651    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521655    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521658    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521663    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521666    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521669    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521672    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521675    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521678    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521681    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521684    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521687    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521691    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521694    2587 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521697    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521699    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:54.278504 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521702    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521705    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521708    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521711    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521713    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521716    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521719    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521722    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521724    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521727    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521730    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521733    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521736    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521740    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521743    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521746    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521749    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521751    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521754    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:54.279137 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521756    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521759    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521761    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521764    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521766    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521769    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521771    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521774    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521777    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521780    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521783    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521786    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521788    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521791    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521794    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521798    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521800    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521803    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521806    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521808    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:54.280018 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521811    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521814    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521816    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521819    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521822    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521824    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521827    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521829    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521832    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521834    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521837    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521839    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521842    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521844    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521847    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.521849    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522251    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522257    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522263    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522266    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:54.280862 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522269    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522272    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522275    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522277    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522280    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522283    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522285    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522288    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522291    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522293    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522296    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522298    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522301    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522303    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522306    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522309    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522311    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522314    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522317    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522319    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:54.281906 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522322    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522325    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522328    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522331    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522333    2587 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522336    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522338    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522341    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522343    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522346    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522350    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522355    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522359    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522363    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522365    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522368    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522372    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522375    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522378    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522381    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:54.282555 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522383    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522386    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522388    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522391    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522393    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522396    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522399    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522401    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522404    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522406    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522409    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522411    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522414    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522416    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522419    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522421    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522424    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522427    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522429    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:54.283184 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522432    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522434    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522437    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522441    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522444    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522446    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522449    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522452    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522454    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522457    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522459    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522462    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522464    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522467    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522469    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522472    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522475    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522477    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522480    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522482    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:54.284021 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522485    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522489    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522492    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522496    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522500    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522503    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522506    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522509    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522511    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522514    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522517    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522520    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522522    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522525    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522527    2587 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.522532    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522607    2587 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522614    2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522625    2587 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522630    2587 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522635    2587 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 18:50:54.285162 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522640    2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522647    2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522651    2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522654    2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522657    2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522660    2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522665    2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522668    2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522671    2587 flags.go:64] FLAG: --cgroup-root=""
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522674    2587 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522677    2587 flags.go:64] FLAG: --client-ca-file=""
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522680    2587 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522684    2587 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522689    2587 flags.go:64] FLAG: --cluster-domain=""
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522692    2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522695    2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522698    2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522701    2587 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522705    2587 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522709    2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522712    2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522715    2587 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522718    2587 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522725    2587 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 18:50:54.286106 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522729    2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522732    2587 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522737    2587 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522741    2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522744    2587 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522747    2587 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522750    2587 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522753    2587 flags.go:64] FLAG: --enable-server="true"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522757    2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522761    2587 flags.go:64] FLAG: --event-burst="100"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522764    2587 flags.go:64] FLAG: --event-qps="50"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522767    2587 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522770    2587 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522773    2587 flags.go:64] FLAG: --eviction-hard=""
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522777    2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522780    2587 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522783    2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522786    2587 flags.go:64] FLAG: --eviction-soft=""
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522789    2587 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522792    2587 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522796    2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522799    2587 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522802    2587 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522805    2587 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522808    2587 flags.go:64] FLAG: --feature-gates=""
Jul 13 18:50:54.286932 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522812    2587 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522815    2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522818    2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522821    2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522824    2587 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522827    2587 flags.go:64] FLAG: --help="false"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522830    2587 flags.go:64] FLAG: --hostname-override="ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522833    2587 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522836    2587 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522840    2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522843    2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522847    2587 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522850    2587 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522852    2587 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522855    2587 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522858    2587 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522862    2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522865    2587 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522868    2587 flags.go:64] FLAG: --kube-reserved=""
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522871    2587 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522874    2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522877    2587 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522879    2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522882    2587 flags.go:64] FLAG: --lock-file=""
Jul 13 18:50:54.287772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522893    2587 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522897    2587 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522901    2587 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522912    2587 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522916    2587 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522920    2587 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522924    2587 flags.go:64] FLAG: --logging-format="text"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522929    2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522934    2587 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522938    2587 flags.go:64] FLAG: --manifest-url=""
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522941    2587 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522947    2587 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522952    2587 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522958    2587 flags.go:64] FLAG: --max-pods="110"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522961    2587 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522966    2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522970    2587 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522976    2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522980    2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522985    2587 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.522989    2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523014    2587 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523019    2587 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523023    2587 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 18:50:54.288781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523028    2587 flags.go:64] FLAG: --pod-cidr=""
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523033    2587 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523041    2587 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523045    2587 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523049    2587 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523054    2587 flags.go:64] FLAG: --port="10250"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523058    2587 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523063    2587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d688b83f8eac64d"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523068    2587 flags.go:64] FLAG: --qos-reserved=""
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523072    2587 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523076    2587 flags.go:64] FLAG: --register-node="true"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523080    2587 flags.go:64] FLAG: --register-with-taints=""
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523087    2587 flags.go:64] FLAG: --registry-burst="10"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523092    2587 flags.go:64] FLAG: --registry-qps="5"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523096    2587 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523101    2587 flags.go:64] FLAG: --reserved-memory=""
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523110    2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523115    2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523120    2587 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523124    2587 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523128    2587 flags.go:64] FLAG: --runonce="false"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523132    2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523137    2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523141    2587 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523145    2587 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 18:50:54.290722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523151    2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523155    2587 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523159    2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523164    2587 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523168    2587 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523173    2587 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523176    2587 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523180    2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523185    2587 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523197    2587 flags.go:64] FLAG: --system-cgroups=""
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523202    2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523210    2587 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523214    2587 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523218    2587 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523228    2587 flags.go:64] FLAG: --tls-min-version=""
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523232    2587 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523236    2587 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523241    2587 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523245    2587 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523252    2587 flags.go:64] FLAG: --v="2"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523258    2587 flags.go:64] FLAG: --version="false"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523264    2587 flags.go:64] FLAG: --vmodule=""
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523269    2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523274    2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523457    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:54.291923 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523463    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523468    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523472    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523476    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523480    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523484    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523488    2587 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523493    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523497    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523501    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523504    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523509    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523512    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523516    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523520    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523525    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523530    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523535    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523539    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523543    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:54.293154 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523546    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523550    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523554    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523558    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523562    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523566    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523571    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523575    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523579    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523583    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523587    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523591    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523593    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523596    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523599    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523603    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523606    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523610    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523614    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523619    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:54.414831 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523623    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523627    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523630    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523636    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523641    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523645    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523650    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523654    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523658    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523664    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523669    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523674    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523678    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523682    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523686    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523690    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523693    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523697    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:54.415814 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523703    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523707    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523710    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523714    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523718    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523721    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523725    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523729    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523733    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523736    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523740    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523743    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523747    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523753    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523757    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523760    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523764    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523768    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523772    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523776    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:54.416429 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523780    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523784    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523787    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523791    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523795    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523800    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523803    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523806    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523809    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523814    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523818    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523822    2587 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523827    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523831    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523835    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523839    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523842    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523846    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523850    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:54.417416 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.523854    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.523861    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.533330    2587 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.533345    2587 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.534060    2587 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.534068    2587 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534140    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534146    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534149    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534152    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534155    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534158    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534161    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534163    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534167    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:54.417964 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534169    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534173    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534178    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534188    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534191    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534193    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534196    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534198    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534201    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534204    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534206    2587 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534211    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534213    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534216    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534219    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534222    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534224    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534227    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534229    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:54.418541 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534233    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534237    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534240    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534242    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534247    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534250    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534252    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534255    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534267    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534270    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534273    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534276    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534278    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534281    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534283    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534286    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534288    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534296    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534299    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:54.419117 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534301    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534304    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534307    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534309    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534312    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534316    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534318    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534321    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534324    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534326    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534329    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534331    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534334    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534336    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534339    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534341    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534344    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534348    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534351    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534354    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:54.419704 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534356    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534358    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534361    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534364    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534366    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534369    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534371    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534374    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534377    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534380    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534382    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534390    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534393    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534395    2587 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534397    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534400    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534402    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534407    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534409    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534412    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:54.558491 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534415    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534417    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534421    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534425    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534428    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534431    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534434    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534437    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534440    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534444    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534447    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534449    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.534455    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534616    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:54.559094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534622    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534625    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534628    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534631    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534634    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534636    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534639    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534643    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534646    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534649    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534658    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534662    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534664    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534667    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534670    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534672    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534676    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534678    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534681    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:54.559545 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534684    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534687    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534689    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534692    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534694    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534697    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534699    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534702    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534706    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534708    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534711    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534714    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534716    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534719    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534721    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534724    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534727    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534729    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534732    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534734    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:54.561094 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534737    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534739    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534742    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534744    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534752    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534755    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534757    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534760    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534763    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534767    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534769    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534772    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534774    2587 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534777    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534780    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534782    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534785    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534788    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534790    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534793    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:54.561725 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534796    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534799    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534802    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534805    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534807    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534810    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534812    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534815    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534817    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534820    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534822    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534825    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534827    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534830    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534832    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534835    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534837    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534845    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534849    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534852    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:54.562368 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534855    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534859    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534862    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534865    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534867    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534870    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534872    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534875    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534877    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534880    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534882    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534892    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534896    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534898    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534901    2587 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534904    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534907    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534909    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:54.563702 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:52.534912    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.534917    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.535053    2587 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.537120    2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.538332    2587 server.go:1019] "Starting client certificate rotation"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.538434    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.538486    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.540795    2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.542678    2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.554701    2587 log.go:25] "Validated CRI v1 runtime API"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.556832    2587 log.go:25] "Validated CRI v1 image API"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:52.557622    2587 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 18:50:54.564219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.240504    2587 fs.go:135] Filesystem UUIDs: map[4aeffdf5-7bca-46cc-b0b1-73bc43f68ff6:/dev/nvme0n1p4 608f1737-0361-4ed3-b82f-5a8f094d3630:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 13 18:50:54.564713 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.240520    2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 18:50:54.564820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.246361    2587 manager.go:216] Machine: {Timestamp:2026-07-13 18:50:54.244485867 +0000 UTC m=+0.484785957 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102443 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27420c91c3707af2ac990dfc60c288 SystemUUID:ec27420c-91c3-707a-f2ac-990dfc60c288 BootID:efb0667c-5b1c-49bb-b03d-6d06a1027a6a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4a:51:3f:ed:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4a:51:3f:ed:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:0e:5e:ab:bc:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 18:50:54.564820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.246469    2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 18:50:54.564820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.246551    2587 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 18:50:54.564820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.249266    2587 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 18:50:54.564820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.249307    2587 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 18:50:54.564820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.250002    2587 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 18:50:54.564820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.250014    2587 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 18:50:54.564820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.250026    2587 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.250597    2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.251878    2587 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.254946    2587 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.255083    2587 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.255773    2587 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.255787    2587 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.255802    2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.255810    2587 kubelet.go:398] "Adding apiserver pod source"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.255819    2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.256251    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.256272    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.258876    2587 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.259765    2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261182    2587 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261226    2587 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261386    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261407    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261413    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261419    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261428    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261434    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261776    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261783    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261790    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261796    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261805    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.261813    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.262382    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.262390    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.266177    2587 server.go:1268] "Started kubelet"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.266754    2587 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.271918    2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.272312    2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.272453    2587 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.273850    2587 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.273759    2587 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.277442    2587 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.279578    2587 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.279615    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 18:50:54.565218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.279629    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.279617    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.279700    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280125    2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280129    2587 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280151    2587 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280408    2587 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280418    2587 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280471    2587 factory.go:55] Registering systemd factory
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280487    2587 factory.go:223] Registration of the systemd container factory successfully
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.279480    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-100.ec2.internal.18c1ee0a07f731fd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-100.ec2.internal,UID:ip-10-0-139-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-100.ec2.internal,},FirstTimestamp:2026-07-13 18:50:54.266151421 +0000 UTC m=+0.506451564,LastTimestamp:2026-07-13 18:50:54.266151421 +0000 UTC m=+0.506451564,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-100.ec2.internal,}"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280794    2587 factory.go:153] Registering CRI-O factory
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280805    2587 factory.go:223] Registration of the crio container factory successfully
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280847    2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280871    2587 factory.go:103] Registering Raw factory
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.280960    2587 manager.go:1196] Started watching for new ooms in manager
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.283675    2587 manager.go:349] Starting recovery of all containers
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.291108    2587 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.292166    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.299880    2587 manager.go:354] Recovery completed
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.301991    2587 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.305867    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.310264    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hdkjh"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.310293    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hdkjh"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.311041    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.311076    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.311089    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.311702    2587 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.311710    2587 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.311724    2587 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:50:54.566281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.311779    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.312525    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.312545    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.312554    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.313131    2587 policy_none.go:49] "None policy: Start"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.313159    2587 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.313377    2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.314437    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-100.ec2.internal.18c1ee0a0aa47975  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-100.ec2.internal,UID:ip-10-0-139-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-100.ec2.internal,},FirstTimestamp:2026-07-13 18:50:54.311061877 +0000 UTC m=+0.551361968,LastTimestamp:2026-07-13 18:50:54.311061877 +0000 UTC m=+0.551361968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-100.ec2.internal,}"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.317977    2587 policy_none.go:47] "Start"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.360782    2587 manager.go:337] "Starting Device Plugin manager"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.361007    2587 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.361018    2587 server.go:87] "Starting device plugin registration server"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.361271    2587 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.361281    2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.361413    2587 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.361517    2587 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.361527    2587 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.362076    2587 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.362126    2587 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.461364    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.462423    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.462455    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.462468    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.462497    2587 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.473000    2587 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.473022    2587 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-100.ec2.internal\": node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.498284    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.499417    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.499464    2587 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.499489    2587 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.499530    2587 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.502444    2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:54.567043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.502648    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.599782    2587 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"]
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.599870    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.600886    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.600916    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.600927    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.605968    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607089    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607224    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607255    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607842    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607862    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607871    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607846    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607934    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.607944    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.609384    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.609408    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.610023    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.610046    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.610056    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.642866    2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.647533    2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.683443    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a676b35a3de8ad1194c9799bed61c8c9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a676b35a3de8ad1194c9799bed61c8c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.683484    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bd969eb5035698eea62499650b843e8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9bd969eb5035698eea62499650b843e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.687548 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.683519    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd969eb5035698eea62499650b843e8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9bd969eb5035698eea62499650b843e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.812098 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.706257    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:54.812098 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.783701    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a676b35a3de8ad1194c9799bed61c8c9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a676b35a3de8ad1194c9799bed61c8c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.812098 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.783729    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bd969eb5035698eea62499650b843e8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9bd969eb5035698eea62499650b843e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.812098 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.783747    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd969eb5035698eea62499650b843e8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9bd969eb5035698eea62499650b843e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.812098 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.783775    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd969eb5035698eea62499650b843e8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9bd969eb5035698eea62499650b843e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.812098 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.783797    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a676b35a3de8ad1194c9799bed61c8c9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a676b35a3de8ad1194c9799bed61c8c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.812098 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.783806    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bd969eb5035698eea62499650b843e8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9bd969eb5035698eea62499650b843e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.812098 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.806965    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:54.959868 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:54.907182    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:54.959868 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.944387    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:54.959868 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:54.951349    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:55.118618 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:55.007802    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:55.118618 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:55.108368    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:55.278801 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.207471    2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:55.278801 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:55.209470    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:55.278801 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.216328    2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 18:50:55.278801 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.216451    2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:50:55.278801 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.216453    2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:50:55.423942 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:55.310275    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:55.423942 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.313409    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 18:45:54 +0000 UTC" deadline="2028-02-27 05:14:58.374674251 +0000 UTC"
Jul 13 18:50:55.423942 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.313433    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14242h24m3.061244029s"
Jul 13 18:50:55.423942 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:55.411029    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Jul 13 18:50:55.541361 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.450312    2587 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:55.541361 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.480657    2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:55.541361 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.494526    2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:50:55.541361 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.496165    2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Jul 13 18:50:55.541361 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.525381    2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:50:55.800385 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.800364    2587 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:55.808508 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:55.808480    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd969eb5035698eea62499650b843e8.slice/crio-bfe7eb29f309c64025b95586047a6ce310c053dfd557b674c7c74d36b0fea61c WatchSource:0}: Error finding container bfe7eb29f309c64025b95586047a6ce310c053dfd557b674c7c74d36b0fea61c: Status 404 returned error can't find the container with id bfe7eb29f309c64025b95586047a6ce310c053dfd557b674c7c74d36b0fea61c
Jul 13 18:50:55.808822 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:55.808807    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda676b35a3de8ad1194c9799bed61c8c9.slice/crio-7434a03438e86311df0c4f232ad04161cd42714eaefc746358ec53648fe23bcf WatchSource:0}: Error finding container 7434a03438e86311df0c4f232ad04161cd42714eaefc746358ec53648fe23bcf: Status 404 returned error can't find the container with id 7434a03438e86311df0c4f232ad04161cd42714eaefc746358ec53648fe23bcf
Jul 13 18:50:55.813851 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:55.813832    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:50:56.257369 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.257337    2587 apiserver.go:52] "Watching apiserver"
Jul 13 18:50:56.266086 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.266059    2587 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 18:50:56.266535 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.266506    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8gk2r","kube-system/konnectivity-agent-94v4l","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw","openshift-cluster-node-tuning-operator/tuned-v5hqb","openshift-dns/node-resolver-nqc2j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","openshift-multus/multus-additional-cni-plugins-dhgp5","openshift-image-registry/node-ca-8fl4m","openshift-multus/multus-5z52j","openshift-multus/network-metrics-daemon-ll8qw","openshift-network-diagnostics/network-check-target-xvd96","openshift-network-operator/iptables-alerter-dh5pz","openshift-ovn-kubernetes/ovnkube-node-7fsmf"]
Jul 13 18:50:56.268136 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.268118    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.269519 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.269495    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.271000 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.270782    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 18:50:56.271000 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.270885    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.271000 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.270907    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 18:50:56.271000 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.270972    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.271261 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.271083    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dzjz4\""
Jul 13 18:50:56.271261 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.270907    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 18:50:56.272069 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.271954    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:56.272069 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.272003    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 18:50:56.272069 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.272009    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hqftg\""
Jul 13 18:50:56.272392 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.272375    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.272636 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.272617    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.273562 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.273503    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4d59l\""
Jul 13 18:50:56.273873 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.273793    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 18:50:56.273873 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.273806    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.274108 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.274087    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 18:50:56.274167 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.274125    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.274701 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.274638    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 18:50:56.274701 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.274687    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 18:50:56.275358 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.275335    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 18:50:56.275456 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.275349    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 18:50:56.275456 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.275377    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 18:50:56.275568 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.275503    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fn6nj\""
Jul 13 18:50:56.275723 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.275639    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.276224 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.276191    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.276700 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.276587    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.276700 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.276658    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:56.277760 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.277742    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.277864 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.277785    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w4p8m\""
Jul 13 18:50:56.278934 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.278500    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 18:50:56.279590 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.279260    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rtbkd\""
Jul 13 18:50:56.279590 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.279301    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.279590 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.279370    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.280608 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.280584    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.280895 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.280873    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 18:50:56.282749 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.281316    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vv6cw\""
Jul 13 18:50:56.282749 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.281427    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 18:50:56.282749 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.282067    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gvlgg\""
Jul 13 18:50:56.282749 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.282123    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 18:50:56.282749 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.282326    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 18:50:56.283050 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.282796    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 18:50:56.284215 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.283732    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:50:56.286404 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.286375    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 18:50:56.286404 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.286404    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mv8w6\""
Jul 13 18:50:56.286655 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.286639    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.286737 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.286681    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 18:50:56.286737 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.286709    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:50:56.287975 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.287956    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:56.288063 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.288027    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:50:56.290300 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.290279    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:50:56.290397 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.290333    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:50:56.296446 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296420    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-cnibin\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.296542 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296459    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.296542 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296485    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60b983b9-9d4d-42fa-b892-465c5104485f-iptables-alerter-script\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.296542 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296511    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-systemd-units\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.296722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296546    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-ovn\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.296722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296572    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1535d30-9a6f-45a5-8f03-f34baaa3ebf4-agent-certs\") pod \"konnectivity-agent-94v4l\" (UID: \"f1535d30-9a6f-45a5-8f03-f34baaa3ebf4\") " pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:50:56.296722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296595    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-hostroot\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.296722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296617    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.296722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296658    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-sys-fs\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.296722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296687    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysconfig\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.296722 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296712    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-slash\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296736    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296763    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovnkube-script-lib\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296788    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-cni-bin\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296821    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-kubelet\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296865    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-device-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296905    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-tuned\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296938    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6g4\" (UniqueName: \"kubernetes.io/projected/4b227090-d946-4bf0-be0f-cba5485fd90f-kube-api-access-dh6g4\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296965    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-system-cni-dir\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.296989    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovnkube-config\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.297013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297012    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-var-lib-kubelet\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297034    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-host\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297059    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297090    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297115    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-log-socket\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297158    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlw29\" (UniqueName: \"kubernetes.io/projected/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-kube-api-access-mlw29\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297182    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8tzj\" (UniqueName: \"kubernetes.io/projected/99124462-a22b-4fec-aa16-5f0512dfaf39-kube-api-access-t8tzj\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297220    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-lib-modules\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297259    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldz27\" (UniqueName: \"kubernetes.io/projected/dff1207e-de9a-45a7-9629-a3e2074a60eb-kube-api-access-ldz27\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297297    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrsc\" (UniqueName: \"kubernetes.io/projected/29c31923-0274-4781-be82-8266a0459019-kube-api-access-dlrsc\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297339    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-var-lib-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297364    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-netns\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297386    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-conf-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297408    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b227090-d946-4bf0-be0f-cba5485fd90f-hosts-file\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.297431 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297429    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-run-netns\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297452    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-cni-bin\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297475    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297509    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1535d30-9a6f-45a5-8f03-f34baaa3ebf4-konnectivity-ca\") pod \"konnectivity-agent-94v4l\" (UID: \"f1535d30-9a6f-45a5-8f03-f34baaa3ebf4\") " pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297532    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-modprobe-d\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297557    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysctl-conf\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297581    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-node-log\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297605    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdcf\" (UniqueName: \"kubernetes.io/projected/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-kube-api-access-fxdcf\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297628    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-registration-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297660    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b227090-d946-4bf0-be0f-cba5485fd90f-tmp-dir\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297695    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-cni-binary-copy\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297733    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-daemon-config\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297755    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-multus-certs\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297782    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-os-release\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.298047 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297805    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297859    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-systemd\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297883    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-env-overrides\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297903    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-socket-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297925    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-etc-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297951    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7146a5f-6f9b-417d-94f3-8aabf7e61501-dbus\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297957    2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297972    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29c31923-0274-4781-be82-8266a0459019-serviceca\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.297995    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-socket-dir-parent\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298012    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298037    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysctl-d\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298052    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52cfk\" (UniqueName: \"kubernetes.io/projected/5bed48f8-1564-4705-9258-755e26ef91cb-kube-api-access-52cfk\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298081    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60b983b9-9d4d-42fa-b892-465c5104485f-host-slash\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298097    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7146a5f-6f9b-417d-94f3-8aabf7e61501-kubelet-config\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.298662 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298120    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-cnibin\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298169    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-sys\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298231    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298254    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z55mg\" (UniqueName: \"kubernetes.io/projected/60b983b9-9d4d-42fa-b892-465c5104485f-kube-api-access-z55mg\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298268    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-cni-netd\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298287    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298305    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-system-cni-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298319    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-cni-multus\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298332    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dff1207e-de9a-45a7-9629-a3e2074a60eb-tmp\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298348    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-kubelet\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298380    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298418    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-os-release\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298447    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-k8s-cni-cncf-io\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298469    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-run\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.299194 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298489    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29c31923-0274-4781-be82-8266a0459019-host\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.299712 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298533    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-cni-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.299712 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298580    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-etc-kubernetes\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.299712 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298602    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-kubernetes\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.299712 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.298624    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-systemd\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.318806 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.318783    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wxpg9"
Jul 13 18:50:56.324956 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.324924    2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:56.328389 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.328360    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wxpg9"
Jul 13 18:50:56.381701 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.381674    2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 18:50:56.399791 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399763    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-run-netns\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.399791 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399794    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-cni-bin\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399812    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399832    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1535d30-9a6f-45a5-8f03-f34baaa3ebf4-konnectivity-ca\") pod \"konnectivity-agent-94v4l\" (UID: \"f1535d30-9a6f-45a5-8f03-f34baaa3ebf4\") " pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399850    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-modprobe-d\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399864    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysctl-conf\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399879    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-run-netns\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399886    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-cni-bin\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399888    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399936    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-node-log\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399957    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdcf\" (UniqueName: \"kubernetes.io/projected/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-kube-api-access-fxdcf\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.399980    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-registration-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.400009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400015    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b227090-d946-4bf0-be0f-cba5485fd90f-tmp-dir\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400021    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-modprobe-d\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400041    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-cni-binary-copy\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400056    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysctl-conf\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400068    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400110    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-node-log\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400129    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-registration-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400135    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-daemon-config\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400312    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-multus-certs\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400350    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-os-release\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400376    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400393    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-multus-certs\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400408    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-systemd\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400435    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-env-overrides\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.400537 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400457    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-socket-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400463    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-systemd\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400490    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-etc-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400513    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7146a5f-6f9b-417d-94f3-8aabf7e61501-dbus\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400535    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29c31923-0274-4781-be82-8266a0459019-serviceca\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400547    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1535d30-9a6f-45a5-8f03-f34baaa3ebf4-konnectivity-ca\") pod \"konnectivity-agent-94v4l\" (UID: \"f1535d30-9a6f-45a5-8f03-f34baaa3ebf4\") " pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400558    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-socket-dir-parent\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400581    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400318    2587 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400606    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysctl-d\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400619    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b227090-d946-4bf0-be0f-cba5485fd90f-tmp-dir\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400671    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2wq\" (UniqueName: \"kubernetes.io/projected/a21fd6a7-5724-4021-928b-f98404324fb8-kube-api-access-zm2wq\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400711    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52cfk\" (UniqueName: \"kubernetes.io/projected/5bed48f8-1564-4705-9258-755e26ef91cb-kube-api-access-52cfk\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400780    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60b983b9-9d4d-42fa-b892-465c5104485f-host-slash\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400802    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7146a5f-6f9b-417d-94f3-8aabf7e61501-dbus\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.401240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400804    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-cni-binary-copy\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400806    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7146a5f-6f9b-417d-94f3-8aabf7e61501-kubelet-config\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400857    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-cnibin\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400864    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-socket-dir-parent\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400876    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-daemon-config\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400882    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-sys\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400929    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-socket-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400940    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-cnibin\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400966    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7146a5f-6f9b-417d-94f3-8aabf7e61501-kubelet-config\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400974    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-etc-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400988    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400992    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-env-overrides\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.400995    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401018    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60b983b9-9d4d-42fa-b892-465c5104485f-host-slash\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401048    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.401908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401054    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-os-release\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401076    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-sys\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401092    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysctl-d\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401078    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z55mg\" (UniqueName: \"kubernetes.io/projected/60b983b9-9d4d-42fa-b892-465c5104485f-kube-api-access-z55mg\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401130    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-cni-netd\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401153    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401173    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-system-cni-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401194    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-cni-multus\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401229    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dff1207e-de9a-45a7-9629-a3e2074a60eb-tmp\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401236    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-cni-netd\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401248    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-kubelet\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401270    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-system-cni-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401285    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-cni-multus\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401271    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401334    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-os-release\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.402601 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401338    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-kubelet\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401375    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.401365    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401409    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-os-release\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401407    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-k8s-cni-cncf-io\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401441    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-k8s-cni-cncf-io\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401451    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-run\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401484    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29c31923-0274-4781-be82-8266a0459019-host\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401489    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-run\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401512    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-cni-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401556    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-etc-kubernetes\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401612    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29c31923-0274-4781-be82-8266a0459019-host\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.401628    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret podName:d7146a5f-6f9b-417d-94f3-8aabf7e61501 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.901604038 +0000 UTC m=+3.141904139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret") pod "global-pull-secret-syncer-8gk2r" (UID: "d7146a5f-6f9b-417d-94f3-8aabf7e61501") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401631    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-etc-kubernetes\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401651    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29c31923-0274-4781-be82-8266a0459019-serviceca\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.403285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401661    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-kubernetes\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.401894    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-cni-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.402415    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.402599    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-systemd\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.402638    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-cnibin\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.402678    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.402707    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60b983b9-9d4d-42fa-b892-465c5104485f-iptables-alerter-script\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.403216    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/60b983b9-9d4d-42fa-b892-465c5104485f-iptables-alerter-script\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.403300    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-systemd\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.403358    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-cnibin\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.403773    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-kubernetes\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.403963 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.403861    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bed48f8-1564-4705-9258-755e26ef91cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404128    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-systemd-units\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404177    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-ovn\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404226    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1535d30-9a6f-45a5-8f03-f34baaa3ebf4-agent-certs\") pod \"konnectivity-agent-94v4l\" (UID: \"f1535d30-9a6f-45a5-8f03-f34baaa3ebf4\") " pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404265    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-hostroot\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404293    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404325    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-sys-fs\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404376    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysconfig\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404423    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-slash\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404450    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.404488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404483    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovnkube-script-lib\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404516    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-cni-bin\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404550    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-kubelet\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404610    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-sys-fs\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404599    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404617    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-kubelet\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404675    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-var-lib-cni-bin\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404696    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404697    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-sysconfig\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404837    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-run-ovn\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404874    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-device-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.404929 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404907    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-tuned\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404938    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6g4\" (UniqueName: \"kubernetes.io/projected/4b227090-d946-4bf0-be0f-cba5485fd90f-kube-api-access-dh6g4\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404976    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dff1207e-de9a-45a7-9629-a3e2074a60eb-tmp\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.404991    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-system-cni-dir\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405032    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-system-cni-dir\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405075    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovnkube-config\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405104    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-systemd-units\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405121    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-var-lib-kubelet\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405141    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-slash\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405165    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-host\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405192    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405241    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405286    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-log-socket\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405294    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.405481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405321    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlw29\" (UniqueName: \"kubernetes.io/projected/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-kube-api-access-mlw29\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405356    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8tzj\" (UniqueName: \"kubernetes.io/projected/99124462-a22b-4fec-aa16-5f0512dfaf39-kube-api-access-t8tzj\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405385    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-lib-modules\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405487    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-log-socket\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405837    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-lib-modules\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405837    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovnkube-script-lib\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405906    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldz27\" (UniqueName: \"kubernetes.io/projected/dff1207e-de9a-45a7-9629-a3e2074a60eb-kube-api-access-ldz27\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405911    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99124462-a22b-4fec-aa16-5f0512dfaf39-device-dir\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405936    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrsc\" (UniqueName: \"kubernetes.io/projected/29c31923-0274-4781-be82-8266a0459019-kube-api-access-dlrsc\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405965    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-var-lib-kubelet\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405970    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-var-lib-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405982    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406004    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-netns\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.405933    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bed48f8-1564-4705-9258-755e26ef91cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.406105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406080    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-hostroot\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.406770 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406134    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dff1207e-de9a-45a7-9629-a3e2074a60eb-host\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.406770 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406171    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-conf-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.406770 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406249    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-host-run-netns\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.406770 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406291    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-var-lib-openvswitch\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.406770 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406330    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b227090-d946-4bf0-be0f-cba5485fd90f-hosts-file\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.406770 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406529    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b227090-d946-4bf0-be0f-cba5485fd90f-hosts-file\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.406770 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406581    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-multus-conf-dir\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.406770 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.406623    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-ovnkube-config\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.408843 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.408623    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1535d30-9a6f-45a5-8f03-f34baaa3ebf4-agent-certs\") pod \"konnectivity-agent-94v4l\" (UID: \"f1535d30-9a6f-45a5-8f03-f34baaa3ebf4\") " pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:50:56.410095 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.410070    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dff1207e-de9a-45a7-9629-a3e2074a60eb-etc-tuned\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.410372 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.410300    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdcf\" (UniqueName: \"kubernetes.io/projected/d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4-kube-api-access-fxdcf\") pod \"multus-5z52j\" (UID: \"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4\") " pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.410515 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.410497    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z55mg\" (UniqueName: \"kubernetes.io/projected/60b983b9-9d4d-42fa-b892-465c5104485f-kube-api-access-z55mg\") pod \"iptables-alerter-dh5pz\" (UID: \"60b983b9-9d4d-42fa-b892-465c5104485f\") " pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.411532 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.411510    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52cfk\" (UniqueName: \"kubernetes.io/projected/5bed48f8-1564-4705-9258-755e26ef91cb-kube-api-access-52cfk\") pod \"multus-additional-cni-plugins-dhgp5\" (UID: \"5bed48f8-1564-4705-9258-755e26ef91cb\") " pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.417109 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.417054    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8tzj\" (UniqueName: \"kubernetes.io/projected/99124462-a22b-4fec-aa16-5f0512dfaf39-kube-api-access-t8tzj\") pod \"aws-ebs-csi-driver-node-6rtjw\" (UID: \"99124462-a22b-4fec-aa16-5f0512dfaf39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.417311 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.417294    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldz27\" (UniqueName: \"kubernetes.io/projected/dff1207e-de9a-45a7-9629-a3e2074a60eb-kube-api-access-ldz27\") pod \"tuned-v5hqb\" (UID: \"dff1207e-de9a-45a7-9629-a3e2074a60eb\") " pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.418739 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.418715    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6g4\" (UniqueName: \"kubernetes.io/projected/4b227090-d946-4bf0-be0f-cba5485fd90f-kube-api-access-dh6g4\") pod \"node-resolver-nqc2j\" (UID: \"4b227090-d946-4bf0-be0f-cba5485fd90f\") " pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.418846 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.418826    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrsc\" (UniqueName: \"kubernetes.io/projected/29c31923-0274-4781-be82-8266a0459019-kube-api-access-dlrsc\") pod \"node-ca-8fl4m\" (UID: \"29c31923-0274-4781-be82-8266a0459019\") " pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.419154 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.419136    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlw29\" (UniqueName: \"kubernetes.io/projected/762f4cb1-637f-40ee-b3ea-2489ccd2f4b9-kube-api-access-mlw29\") pod \"ovnkube-node-7fsmf\" (UID: \"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.504761 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.504663    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"a676b35a3de8ad1194c9799bed61c8c9","Type":"ContainerStarted","Data":"7434a03438e86311df0c4f232ad04161cd42714eaefc746358ec53648fe23bcf"}
Jul 13 18:50:56.506139 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.506110    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"9bd969eb5035698eea62499650b843e8","Type":"ContainerStarted","Data":"bfe7eb29f309c64025b95586047a6ce310c053dfd557b674c7c74d36b0fea61c"}
Jul 13 18:50:56.507490 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.507408    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:50:56.507490 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.507471    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:56.507659 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.507502    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2wq\" (UniqueName: \"kubernetes.io/projected/a21fd6a7-5724-4021-928b-f98404324fb8-kube-api-access-zm2wq\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:56.507659 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.507619    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:56.507750 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.507733    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs podName:a21fd6a7-5724-4021-928b-f98404324fb8 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:57.007710526 +0000 UTC m=+3.248010618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs") pod "network-metrics-daemon-ll8qw" (UID: "a21fd6a7-5724-4021-928b-f98404324fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:56.518772 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.518748    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:56.518772 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.518775    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:56.518963 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.518791    2587 projected.go:196] Error preparing data for projected volume kube-api-access-jgcjz for pod openshift-network-diagnostics/network-check-target-xvd96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:56.518963 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.518872    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz podName:a95bbd20-fbdc-48c9-bed2-2341f164d5e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:57.018852691 +0000 UTC m=+3.259152779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jgcjz" (UniqueName: "kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz") pod "network-check-target-xvd96" (UID: "a95bbd20-fbdc-48c9-bed2-2341f164d5e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:56.521090 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.521057    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2wq\" (UniqueName: \"kubernetes.io/projected/a21fd6a7-5724-4021-928b-f98404324fb8-kube-api-access-zm2wq\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:56.585040 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.585003    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5z52j"
Jul 13 18:50:56.590812 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.590789    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dh5pz"
Jul 13 18:50:56.599127 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.599101    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:50:56.604995 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.604969    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw"
Jul 13 18:50:56.611847 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.611829    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v5hqb"
Jul 13 18:50:56.619217 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.619185    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqc2j"
Jul 13 18:50:56.626967 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.626948    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8fl4m"
Jul 13 18:50:56.634695 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.634677    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dhgp5"
Jul 13 18:50:56.639387 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.639369    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:50:56.902841 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:56.902816    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99124462_a22b_4fec_aa16_5f0512dfaf39.slice/crio-7c440452abc7e8e0ea8f04a0063ef974278f1743c2ccfa4552327ce7fba1e555 WatchSource:0}: Error finding container 7c440452abc7e8e0ea8f04a0063ef974278f1743c2ccfa4552327ce7fba1e555: Status 404 returned error can't find the container with id 7c440452abc7e8e0ea8f04a0063ef974278f1743c2ccfa4552327ce7fba1e555
Jul 13 18:50:56.903479 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:56.903428    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bed48f8_1564_4705_9258_755e26ef91cb.slice/crio-19b2c49dfa74618e1aa84f50881bc1deb5e005f12f39133ea0a781ce6d322bd6 WatchSource:0}: Error finding container 19b2c49dfa74618e1aa84f50881bc1deb5e005f12f39133ea0a781ce6d322bd6: Status 404 returned error can't find the container with id 19b2c49dfa74618e1aa84f50881bc1deb5e005f12f39133ea0a781ce6d322bd6
Jul 13 18:50:56.904235 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:56.904186    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762f4cb1_637f_40ee_b3ea_2489ccd2f4b9.slice/crio-72939d527e9e46d196944ec4e2dd2e0a516c62c75e7f495a2b5e53acbd4fe98f WatchSource:0}: Error finding container 72939d527e9e46d196944ec4e2dd2e0a516c62c75e7f495a2b5e53acbd4fe98f: Status 404 returned error can't find the container with id 72939d527e9e46d196944ec4e2dd2e0a516c62c75e7f495a2b5e53acbd4fe98f
Jul 13 18:50:56.905910 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:56.905885    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f3c2f9_e9bb_4fd4_bdbb_d14f3151d9f4.slice/crio-9315f13274f2dfb6da35fb3ff8fea1c9053ed4427374767818e9492d147a1b50 WatchSource:0}: Error finding container 9315f13274f2dfb6da35fb3ff8fea1c9053ed4427374767818e9492d147a1b50: Status 404 returned error can't find the container with id 9315f13274f2dfb6da35fb3ff8fea1c9053ed4427374767818e9492d147a1b50
Jul 13 18:50:56.906910 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:56.906562    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b227090_d946_4bf0_be0f_cba5485fd90f.slice/crio-8a33e078b39c8c648add93564ae184f63ff726aaa649145e1c041b2cc30188ea WatchSource:0}: Error finding container 8a33e078b39c8c648add93564ae184f63ff726aaa649145e1c041b2cc30188ea: Status 404 returned error can't find the container with id 8a33e078b39c8c648add93564ae184f63ff726aaa649145e1c041b2cc30188ea
Jul 13 18:50:56.908187 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:56.907944    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1535d30_9a6f_45a5_8f03_f34baaa3ebf4.slice/crio-a5b4a57e7d18952dff5181b9701963f1c5e6b5b2faee21cebbc4fa7f0076d892 WatchSource:0}: Error finding container a5b4a57e7d18952dff5181b9701963f1c5e6b5b2faee21cebbc4fa7f0076d892: Status 404 returned error can't find the container with id a5b4a57e7d18952dff5181b9701963f1c5e6b5b2faee21cebbc4fa7f0076d892
Jul 13 18:50:56.909043 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:56.908726    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff1207e_de9a_45a7_9629_a3e2074a60eb.slice/crio-ffff06017b20c6c0aeb7e6ee28759aafd07489259790149a20a52d73195276d4 WatchSource:0}: Error finding container ffff06017b20c6c0aeb7e6ee28759aafd07489259790149a20a52d73195276d4: Status 404 returned error can't find the container with id ffff06017b20c6c0aeb7e6ee28759aafd07489259790149a20a52d73195276d4
Jul 13 18:50:56.910802 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:56.910770    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:56.910933 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.910914    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:56.911025 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:56.910973    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret podName:d7146a5f-6f9b-417d-94f3-8aabf7e61501 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:57.910955707 +0000 UTC m=+4.151255801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret") pod "global-pull-secret-syncer-8gk2r" (UID: "d7146a5f-6f9b-417d-94f3-8aabf7e61501") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:56.911170 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:50:56.911060    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c31923_0274_4781_be82_8266a0459019.slice/crio-65924b27dc4830ba7d3826efbb15a97fd7aeb9a069c95c459f8eb27dfeb8eb2d WatchSource:0}: Error finding container 65924b27dc4830ba7d3826efbb15a97fd7aeb9a069c95c459f8eb27dfeb8eb2d: Status 404 returned error can't find the container with id 65924b27dc4830ba7d3826efbb15a97fd7aeb9a069c95c459f8eb27dfeb8eb2d
Jul 13 18:50:57.011085 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.011055    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:57.011256 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.011194    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:57.011319 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.011297    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs podName:a21fd6a7-5724-4021-928b-f98404324fb8 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:58.011257119 +0000 UTC m=+4.251557197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs") pod "network-metrics-daemon-ll8qw" (UID: "a21fd6a7-5724-4021-928b-f98404324fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:57.112003 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.111972    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:50:57.112189 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.112167    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:57.112247 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.112212    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:57.112247 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.112229    2587 projected.go:196] Error preparing data for projected volume kube-api-access-jgcjz for pod openshift-network-diagnostics/network-check-target-xvd96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:57.112314 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.112300    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz podName:a95bbd20-fbdc-48c9-bed2-2341f164d5e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:58.112279872 +0000 UTC m=+4.352579972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgcjz" (UniqueName: "kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz") pod "network-check-target-xvd96" (UID: "a95bbd20-fbdc-48c9-bed2-2341f164d5e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:57.331195 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.331118    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:45:56 +0000 UTC" deadline="2028-02-25 07:26:24.23429791 +0000 UTC"
Jul 13 18:50:57.331195 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.331188    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14196h35m26.903113775s"
Jul 13 18:50:57.500924 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.500317    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:57.500924 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.500666    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:50:57.515228 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.514753    2587 generic.go:356] "Generic (PLEG): container finished" podID="9bd969eb5035698eea62499650b843e8" containerID="c0cd6d75d18e70eb4a6c031c1e793b4da5b8dc5b13125c4cd8bc817946239f6c" exitCode=0
Jul 13 18:50:57.515228 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.515039    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"9bd969eb5035698eea62499650b843e8","Type":"ContainerDied","Data":"c0cd6d75d18e70eb4a6c031c1e793b4da5b8dc5b13125c4cd8bc817946239f6c"}
Jul 13 18:50:57.517317 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.517266    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-94v4l" event={"ID":"f1535d30-9a6f-45a5-8f03-f34baaa3ebf4","Type":"ContainerStarted","Data":"a5b4a57e7d18952dff5181b9701963f1c5e6b5b2faee21cebbc4fa7f0076d892"}
Jul 13 18:50:57.519160 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.519122    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqc2j" event={"ID":"4b227090-d946-4bf0-be0f-cba5485fd90f","Type":"ContainerStarted","Data":"8a33e078b39c8c648add93564ae184f63ff726aaa649145e1c041b2cc30188ea"}
Jul 13 18:50:57.520644 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.520614    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5z52j" event={"ID":"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4","Type":"ContainerStarted","Data":"9315f13274f2dfb6da35fb3ff8fea1c9053ed4427374767818e9492d147a1b50"}
Jul 13 18:50:57.523099 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.523014    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" event={"ID":"5bed48f8-1564-4705-9258-755e26ef91cb","Type":"ContainerStarted","Data":"19b2c49dfa74618e1aa84f50881bc1deb5e005f12f39133ea0a781ce6d322bd6"}
Jul 13 18:50:57.526115 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.526087    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dh5pz" event={"ID":"60b983b9-9d4d-42fa-b892-465c5104485f","Type":"ContainerStarted","Data":"955ab1f604962f898accd5a4e10dc11e9636669b225f7d4bef93bd0a9047d055"}
Jul 13 18:50:57.527558 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.527496    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8fl4m" event={"ID":"29c31923-0274-4781-be82-8266a0459019","Type":"ContainerStarted","Data":"65924b27dc4830ba7d3826efbb15a97fd7aeb9a069c95c459f8eb27dfeb8eb2d"}
Jul 13 18:50:57.530087 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.530021    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v5hqb" event={"ID":"dff1207e-de9a-45a7-9629-a3e2074a60eb","Type":"ContainerStarted","Data":"ffff06017b20c6c0aeb7e6ee28759aafd07489259790149a20a52d73195276d4"}
Jul 13 18:50:57.532593 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.532499    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"72939d527e9e46d196944ec4e2dd2e0a516c62c75e7f495a2b5e53acbd4fe98f"}
Jul 13 18:50:57.535652 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.535574    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw" event={"ID":"99124462-a22b-4fec-aa16-5f0512dfaf39","Type":"ContainerStarted","Data":"7c440452abc7e8e0ea8f04a0063ef974278f1743c2ccfa4552327ce7fba1e555"}
Jul 13 18:50:57.919159 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:57.919063    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:57.919350 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.919250    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:57.919350 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:57.919322    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret podName:d7146a5f-6f9b-417d-94f3-8aabf7e61501 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:59.919300867 +0000 UTC m=+6.159600957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret") pod "global-pull-secret-syncer-8gk2r" (UID: "d7146a5f-6f9b-417d-94f3-8aabf7e61501") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:58.020337 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:58.020300    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:58.020515 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:58.020483    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:58.020579 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:58.020548    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs podName:a21fd6a7-5724-4021-928b-f98404324fb8 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:00.020529362 +0000 UTC m=+6.260829443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs") pod "network-metrics-daemon-ll8qw" (UID: "a21fd6a7-5724-4021-928b-f98404324fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:58.121061 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:58.121013    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:50:58.121260 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:58.121239    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:58.121260 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:58.121261    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:58.121388 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:58.121273    2587 projected.go:196] Error preparing data for projected volume kube-api-access-jgcjz for pod openshift-network-diagnostics/network-check-target-xvd96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:58.121388 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:58.121332    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz podName:a95bbd20-fbdc-48c9-bed2-2341f164d5e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:00.121314938 +0000 UTC m=+6.361615021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgcjz" (UniqueName: "kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz") pod "network-check-target-xvd96" (UID: "a95bbd20-fbdc-48c9-bed2-2341f164d5e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:58.286101 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:58.286015    2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:58.334483 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:58.332287    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:45:56 +0000 UTC" deadline="2028-04-06 17:22:02.408040432 +0000 UTC"
Jul 13 18:50:58.334483 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:58.332324    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15190h31m4.075720466s"
Jul 13 18:50:58.500352 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:58.500321    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:58.500525 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:58.500456    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:50:58.503255 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:58.503087    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:50:58.503255 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:58.503191    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:50:58.545992 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:58.545907    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"9bd969eb5035698eea62499650b843e8","Type":"ContainerStarted","Data":"e73386905edb2e7b052cef35a2f9c7e9b2fc7eab0e57af6630ae537e29467e3b"}
Jul 13 18:50:59.500248 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:59.500210    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:50:59.500753 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:59.500359    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:50:59.936187 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:50:59.936091    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:50:59.936366 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:59.936244    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:59.936366 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:50:59.936309    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret podName:d7146a5f-6f9b-417d-94f3-8aabf7e61501 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.936290892 +0000 UTC m=+10.176590972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret") pod "global-pull-secret-syncer-8gk2r" (UID: "d7146a5f-6f9b-417d-94f3-8aabf7e61501") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:51:00.037164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:00.037126    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:00.037384 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:00.037315    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:51:00.037457 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:00.037389    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs podName:a21fd6a7-5724-4021-928b-f98404324fb8 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:04.037369247 +0000 UTC m=+10.277669324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs") pod "network-metrics-daemon-ll8qw" (UID: "a21fd6a7-5724-4021-928b-f98404324fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:51:00.138443 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:00.138404    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:00.138634 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:00.138618    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:51:00.138692 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:00.138641    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:51:00.138692 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:00.138653    2587 projected.go:196] Error preparing data for projected volume kube-api-access-jgcjz for pod openshift-network-diagnostics/network-check-target-xvd96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:51:00.138838 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:00.138720    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz podName:a95bbd20-fbdc-48c9-bed2-2341f164d5e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:04.138702466 +0000 UTC m=+10.379002545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgcjz" (UniqueName: "kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz") pod "network-check-target-xvd96" (UID: "a95bbd20-fbdc-48c9-bed2-2341f164d5e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:51:00.500515 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:00.500479    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:00.501042 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:00.500626    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:00.501042 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:00.500822    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:00.501042 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:00.500913    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:01.500177 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:01.500138    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:01.500402 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:01.500309    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:02.500585 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:02.500123    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:02.500585 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:02.500171    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:02.500585 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:02.500281    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:02.500585 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:02.500416    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:03.499941 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:03.499905    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:03.500143 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:03.500067    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:03.973462 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:03.972777    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:03.973462 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:03.972931    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:51:03.973462 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:03.973022    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret podName:d7146a5f-6f9b-417d-94f3-8aabf7e61501 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:11.972989157 +0000 UTC m=+18.213289257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret") pod "global-pull-secret-syncer-8gk2r" (UID: "d7146a5f-6f9b-417d-94f3-8aabf7e61501") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:51:04.074325 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:04.073570    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:04.074325 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:04.073842    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:51:04.074325 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:04.073912    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs podName:a21fd6a7-5724-4021-928b-f98404324fb8 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:12.073891143 +0000 UTC m=+18.314191225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs") pod "network-metrics-daemon-ll8qw" (UID: "a21fd6a7-5724-4021-928b-f98404324fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:51:04.174442 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:04.174364    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:04.174638 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:04.174531    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:51:04.174638 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:04.174558    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:51:04.174638 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:04.174573    2587 projected.go:196] Error preparing data for projected volume kube-api-access-jgcjz for pod openshift-network-diagnostics/network-check-target-xvd96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:51:04.174804 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:04.174647    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz podName:a95bbd20-fbdc-48c9-bed2-2341f164d5e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:12.174625489 +0000 UTC m=+18.414925570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgcjz" (UniqueName: "kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz") pod "network-check-target-xvd96" (UID: "a95bbd20-fbdc-48c9-bed2-2341f164d5e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:51:04.501671 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:04.501169    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:04.501671 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:04.501305    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:04.501937 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:04.501768    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:04.501937 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:04.501871    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:05.501050 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:05.500528    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:05.501050 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:05.500676    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:06.500306 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:06.500245    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:06.500306 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:06.500271    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:06.500551 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:06.500442    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:06.500551 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:06.500542    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:07.499851 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:07.499818    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:07.500227 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:07.499968    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:08.500309 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:08.500281    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:08.500309 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:08.500315    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:08.500823 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:08.500414    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:08.500823 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:08.500529    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:09.500624 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:09.500588    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:09.501116 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:09.500734    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:10.500178 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:10.500140    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:10.500377 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:10.500292    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:10.500377 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:10.500336    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:10.500489 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:10.500454    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:11.500327 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:11.500288    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:11.500812 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:11.500414    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:12.039171 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:12.039136    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:12.039390 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.039289    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:51:12.039390 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.039357    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret podName:d7146a5f-6f9b-417d-94f3-8aabf7e61501 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:28.039339299 +0000 UTC m=+34.279639379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret") pod "global-pull-secret-syncer-8gk2r" (UID: "d7146a5f-6f9b-417d-94f3-8aabf7e61501") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:51:12.139805 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:12.139762    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:12.139979 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.139925    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:51:12.140024 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.140007    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs podName:a21fd6a7-5724-4021-928b-f98404324fb8 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:28.139989258 +0000 UTC m=+34.380289337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs") pod "network-metrics-daemon-ll8qw" (UID: "a21fd6a7-5724-4021-928b-f98404324fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:51:12.240462 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:12.240412    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:12.240634 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.240569    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:51:12.240634 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.240593    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:51:12.240634 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.240604    2587 projected.go:196] Error preparing data for projected volume kube-api-access-jgcjz for pod openshift-network-diagnostics/network-check-target-xvd96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:51:12.240763 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.240659    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz podName:a95bbd20-fbdc-48c9-bed2-2341f164d5e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:28.240642483 +0000 UTC m=+34.480942566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgcjz" (UniqueName: "kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz") pod "network-check-target-xvd96" (UID: "a95bbd20-fbdc-48c9-bed2-2341f164d5e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:51:12.499775 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:12.499692    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:12.499924 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.499832    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:12.499924 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:12.499874    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:12.499999 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:12.499959    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:13.500407 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:13.500372    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:13.500928 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:13.500509    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:14.504355 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:14.503400    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:14.504355 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:14.503625    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:14.504355 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:14.504001    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:14.504355 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:14.504163    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:15.500619 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:15.500585    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:15.500800 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:15.500713    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:16.499977 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:16.499939    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:16.500378 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:16.499939    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:16.500378 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:16.500055    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:16.500378 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:16.500110    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:17.500449 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.500090    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:17.501138 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:17.500576    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:17.593351 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.593317    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8fl4m" event={"ID":"29c31923-0274-4781-be82-8266a0459019","Type":"ContainerStarted","Data":"fe29cd8bdf3709c0023b1b265b14ee3e427c94597d5e0b8766823a5c842ab73d"}
Jul 13 18:51:17.594805 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.594776    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v5hqb" event={"ID":"dff1207e-de9a-45a7-9629-a3e2074a60eb","Type":"ContainerStarted","Data":"51f294db7bc7186ca74d2308b7e61eeb46885899ee2a33a3d654991e1dff00a5"}
Jul 13 18:51:17.597465 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.597447    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 18:51:17.597825 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.597805    2587 generic.go:356] "Generic (PLEG): container finished" podID="762f4cb1-637f-40ee-b3ea-2489ccd2f4b9" containerID="88345cf479aec80889e679aba33ab4f87f10dc68e41a3b9af5ab0581ab4084c1" exitCode=1
Jul 13 18:51:17.597901 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.597871    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"76afeef8b777ec87809b89b45b46226c8d1c7818794b17b706c12f0543a7c7bc"}
Jul 13 18:51:17.597953 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.597930    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"682b4e397955ca60dd09d3ffab74ffb24d6f39ed535dae21ca4bfb741fd8656f"}
Jul 13 18:51:17.597996 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.597979    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"b32c26c5927f13b3b3feb9bc6fb3894f22f27cd9ff428b185c018f84a7f59aea"}
Jul 13 18:51:17.598042 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.598014    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"970045aff3246c4d279ef519d42e09669b5236fe7bcba1e27b33d0e060f25b89"}
Jul 13 18:51:17.598081 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.598049    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerDied","Data":"88345cf479aec80889e679aba33ab4f87f10dc68e41a3b9af5ab0581ab4084c1"}
Jul 13 18:51:17.598081 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.598079    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"4760599a8cab0e5a180874908b0d6824e5439cb1ae075e2165cf9f911fadd4c8"}
Jul 13 18:51:17.599295 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.599271    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw" event={"ID":"99124462-a22b-4fec-aa16-5f0512dfaf39","Type":"ContainerStarted","Data":"c50251e2ff7a490430662e04b317d82b22de96d5d795d2453fa2735759179c30"}
Jul 13 18:51:17.600691 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.600671    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"a676b35a3de8ad1194c9799bed61c8c9","Type":"ContainerStarted","Data":"b590541184651789e6923aa9dd82293462afba94acfffb71a7d15d1f044e346d"}
Jul 13 18:51:17.602173 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.602142    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-94v4l" event={"ID":"f1535d30-9a6f-45a5-8f03-f34baaa3ebf4","Type":"ContainerStarted","Data":"2b8da2507f189c9c940d640212cdc8ad830a04b366404cb9d6a619b7476c2a83"}
Jul 13 18:51:17.603508 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.603486    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqc2j" event={"ID":"4b227090-d946-4bf0-be0f-cba5485fd90f","Type":"ContainerStarted","Data":"15510ef4e9e479e5a58efdecafd5f3e367c1307120ddb94da5ad041b5d73817e"}
Jul 13 18:51:17.604995 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.604965    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5z52j" event={"ID":"d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4","Type":"ContainerStarted","Data":"f858fcbfa5f70bbe7445fa724380ef581c4c65def6be5452eda4123b300bed06"}
Jul 13 18:51:17.606426 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.606403    2587 generic.go:356] "Generic (PLEG): container finished" podID="5bed48f8-1564-4705-9258-755e26ef91cb" containerID="0e94d80d638698fc69aefcac4c2f7d4c82f0c79ad99f87a9e3cf61c5acfd4eae" exitCode=0
Jul 13 18:51:17.606511 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.606438    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" event={"ID":"5bed48f8-1564-4705-9258-755e26ef91cb","Type":"ContainerDied","Data":"0e94d80d638698fc69aefcac4c2f7d4c82f0c79ad99f87a9e3cf61c5acfd4eae"}
Jul 13 18:51:17.609855 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.609806    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" podStartSLOduration=22.609791273 podStartE2EDuration="22.609791273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:58.565562361 +0000 UTC m=+4.805862462" watchObservedRunningTime="2026-07-13 18:51:17.609791273 +0000 UTC m=+23.850091373"
Jul 13 18:51:17.610022 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.609996    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8fl4m" podStartSLOduration=3.89303873 podStartE2EDuration="23.609988861s" totalImagesPullingTime="19.716950131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:17.60974535 +0000 UTC m=+23.850045450" watchObservedRunningTime="2026-07-13 18:51:17.609988861 +0000 UTC m=+23.850288961"
Jul 13 18:51:17.630616 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.630569    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5z52j" podStartSLOduration=3.8696393159999998 podStartE2EDuration="23.630553449s" totalImagesPullingTime="19.760914133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:17.630479933 +0000 UTC m=+23.870780032" watchObservedRunningTime="2026-07-13 18:51:17.630553449 +0000 UTC m=+23.870853549"
Jul 13 18:51:17.671405 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.671343    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-94v4l" podStartSLOduration=3.951965335 podStartE2EDuration="23.671320801s" totalImagesPullingTime="19.719355466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:17.65035793 +0000 UTC m=+23.890658031" watchObservedRunningTime="2026-07-13 18:51:17.671320801 +0000 UTC m=+23.911620907"
Jul 13 18:51:17.671605 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.671577    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v5hqb" podStartSLOduration=3.9410033 podStartE2EDuration="23.671566768s" totalImagesPullingTime="19.730563468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:17.670882327 +0000 UTC m=+23.911182428" watchObservedRunningTime="2026-07-13 18:51:17.671566768 +0000 UTC m=+23.911866870"
Jul 13 18:51:17.687281 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.687227    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nqc2j" podStartSLOduration=3.949119183 podStartE2EDuration="23.687187592s" totalImagesPullingTime="19.738068409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:17.686648633 +0000 UTC m=+23.926948733" watchObservedRunningTime="2026-07-13 18:51:17.687187592 +0000 UTC m=+23.927487696"
Jul 13 18:51:17.702190 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.702154    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:17.726212 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.726159    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" podStartSLOduration=22.726143923 podStartE2EDuration="22.726143923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:17.725787597 +0000 UTC m=+23.966087697" watchObservedRunningTime="2026-07-13 18:51:17.726143923 +0000 UTC m=+23.966444023"
Jul 13 18:51:17.779723 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.779653    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:17.875983 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.875955    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:17.977465 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:17.977428    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:18.115074 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.115047    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:51:18.115630 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.115611    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:51:18.119820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.119797    2587 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 18:51:18.376356 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.376168    2587 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T18:51:18.119813146Z","UUID":"a5278960-ef4a-4b06-ba40-7c611c048506","Handler":null,"Name":"","Endpoint":""}
Jul 13 18:51:18.378020 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.377996    2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 18:51:18.378160 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.378044    2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 18:51:18.500806 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.500772    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:18.501244 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.500772    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:18.501244 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:18.500913    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:18.501244 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:18.500951    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:18.610567 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.610528    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dh5pz" event={"ID":"60b983b9-9d4d-42fa-b892-465c5104485f","Type":"ContainerStarted","Data":"cdb5c46498565638acf11d6e0bb57a5795ee49f56aeb52c5431a93a815cd7df7"}
Jul 13 18:51:18.612779 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.612738    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw" event={"ID":"99124462-a22b-4fec-aa16-5f0512dfaf39","Type":"ContainerStarted","Data":"ccd0b1e2b9393bbb9df0355257c70063889680ab32915eb76ca5efcc599308f4"}
Jul 13 18:51:18.627494 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.627393    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dh5pz" podStartSLOduration=4.883441505 podStartE2EDuration="24.627373974s" totalImagesPullingTime="19.743932469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:18.626610722 +0000 UTC m=+24.866910833" watchObservedRunningTime="2026-07-13 18:51:18.627373974 +0000 UTC m=+24.867674074"
Jul 13 18:51:18.750326 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.750257    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:18.811358 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.811319    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:18.986724 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:18.986619    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:19.047040 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:19.047000    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:19.500457 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:19.500424    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:19.500632 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:19.500547    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:19.617493 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:19.617375    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw" event={"ID":"99124462-a22b-4fec-aa16-5f0512dfaf39","Type":"ContainerStarted","Data":"4d12a08d61f624fe87dd251b10a44be55402423ea80d157718b90d9cbee989ca"}
Jul 13 18:51:19.617493 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:19.617436    2587 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 13 18:51:20.500737 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:20.500457    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:20.500945 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:20.500521    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:20.500945 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:20.500847    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:20.500945 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:20.500921    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:20.623358 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:20.623330    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 18:51:20.623830 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:20.623723    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"27a58f236d76a5f9f4e7815efaf5ab71f77819233b558a6dfae21372eeaf2716"}
Jul 13 18:51:21.499873 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:21.499834    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:21.500123 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:21.499992    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:22.500132 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.499850    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:22.500132 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.499871    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:22.500900 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:22.500161    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:22.500900 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:22.500221    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:22.632316 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.632287    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 18:51:22.632687 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.632651    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"4c973eab59fee6d331a632c2ecb7af9deef8e356ca708af2cc5cb84e6c87f770"}
Jul 13 18:51:22.632954 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.632923    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:51:22.633181 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.633164    2587 scope.go:117] "RemoveContainer" containerID="88345cf479aec80889e679aba33ab4f87f10dc68e41a3b9af5ab0581ab4084c1"
Jul 13 18:51:22.634785 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.634760    2587 generic.go:356] "Generic (PLEG): container finished" podID="5bed48f8-1564-4705-9258-755e26ef91cb" containerID="b38cae552a2d141d8cb3c96b3b5ff8d544d6e7e03bb2c2c94774af62c3499077" exitCode=0
Jul 13 18:51:22.634879 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.634814    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" event={"ID":"5bed48f8-1564-4705-9258-755e26ef91cb","Type":"ContainerDied","Data":"b38cae552a2d141d8cb3c96b3b5ff8d544d6e7e03bb2c2c94774af62c3499077"}
Jul 13 18:51:22.650115 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.650085    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:51:22.662648 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:22.662599    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rtjw" podStartSLOduration=6.567631223 podStartE2EDuration="28.662584316s" totalImagesPullingTime="22.094953093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:19.644545947 +0000 UTC m=+25.884846047" watchObservedRunningTime="2026-07-13 18:51:22.662584316 +0000 UTC m=+28.902884420"
Jul 13 18:51:23.067054 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.067017    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:51:23.067253 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.067138    2587 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 13 18:51:23.067609 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.067592    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-94v4l"
Jul 13 18:51:23.180495 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.180457    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:23.500169 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.500091    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:23.500614 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:23.500223    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:23.640326 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.640300    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 18:51:23.640676 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.640656    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" event={"ID":"762f4cb1-637f-40ee-b3ea-2489ccd2f4b9","Type":"ContainerStarted","Data":"64bf6e607193573a7e520d2690ecbede87d816bf414792f4efa0006e875c0263"}
Jul 13 18:51:23.641054 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.641012    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:51:23.641156 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.641098    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:51:23.658165 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.658137    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:51:23.675747 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.675679    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf" podStartSLOduration=9.709019555 podStartE2EDuration="29.675662996s" totalImagesPullingTime="19.966643441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:23.674166592 +0000 UTC m=+29.914466684" watchObservedRunningTime="2026-07-13 18:51:23.675662996 +0000 UTC m=+29.915963087"
Jul 13 18:51:23.847011 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.846917    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8gk2r"]
Jul 13 18:51:23.847170 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.847049    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:23.847170 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:23.847145    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:23.852543 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.852493    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xvd96"]
Jul 13 18:51:23.852731 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.852637    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:23.852799 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:23.852752    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:23.853140 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.853115    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ll8qw"]
Jul 13 18:51:23.853259 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:23.853247    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:23.853392 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:23.853364    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:25.499958 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:25.499692    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:25.500398 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:25.499740    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:25.500398 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:25.499981    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:25.500398 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:25.500068    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:25.500398 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:25.499740    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:25.500398 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:25.500181    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:25.646795 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:25.646756    2587 generic.go:356] "Generic (PLEG): container finished" podID="5bed48f8-1564-4705-9258-755e26ef91cb" containerID="1e4b08ca3b13acd24bac97849bdd5628d740a326a510abe7e5f78171d25a1e51" exitCode=0
Jul 13 18:51:25.646952 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:25.646817    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" event={"ID":"5bed48f8-1564-4705-9258-755e26ef91cb","Type":"ContainerDied","Data":"1e4b08ca3b13acd24bac97849bdd5628d740a326a510abe7e5f78171d25a1e51"}
Jul 13 18:51:27.500246 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:27.500190    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:27.501010 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:27.500190    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:27.501010 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:27.500337    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:27.501010 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:27.500190    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:27.501010 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:27.500393    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:27.501010 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:27.500477    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:27.654101 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:27.654070    2587 generic.go:356] "Generic (PLEG): container finished" podID="5bed48f8-1564-4705-9258-755e26ef91cb" containerID="23fd018d9fac61c95770ab353b3c0985e46b3ac484f559904a4f48b5d2372838" exitCode=0
Jul 13 18:51:27.654282 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:27.654111    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" event={"ID":"5bed48f8-1564-4705-9258-755e26ef91cb","Type":"ContainerDied","Data":"23fd018d9fac61c95770ab353b3c0985e46b3ac484f559904a4f48b5d2372838"}
Jul 13 18:51:28.061192 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:28.061133    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:28.061399 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:28.061307    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:51:28.061399 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:28.061382    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret podName:d7146a5f-6f9b-417d-94f3-8aabf7e61501 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:00.061366666 +0000 UTC m=+66.301666749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret") pod "global-pull-secret-syncer-8gk2r" (UID: "d7146a5f-6f9b-417d-94f3-8aabf7e61501") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:51:28.161978 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:28.161943    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:28.162155 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:28.162069    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:51:28.162155 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:28.162124    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs podName:a21fd6a7-5724-4021-928b-f98404324fb8 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:00.162110746 +0000 UTC m=+66.402410824 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs") pod "network-metrics-daemon-ll8qw" (UID: "a21fd6a7-5724-4021-928b-f98404324fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:51:28.263000 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:28.262948    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:28.263309 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:28.263119    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:51:28.263309 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:28.263139    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:51:28.263309 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:28.263149    2587 projected.go:196] Error preparing data for projected volume kube-api-access-jgcjz for pod openshift-network-diagnostics/network-check-target-xvd96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:51:28.263309 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:28.263231    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz podName:a95bbd20-fbdc-48c9-bed2-2341f164d5e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:00.263195456 +0000 UTC m=+66.503495534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgcjz" (UniqueName: "kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz") pod "network-check-target-xvd96" (UID: "a95bbd20-fbdc-48c9-bed2-2341f164d5e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:51:29.500437 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.500357    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:29.501026 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.500357    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:29.501026 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:29.500506    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gk2r" podUID="d7146a5f-6f9b-417d-94f3-8aabf7e61501"
Jul 13 18:51:29.501026 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.500355    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:29.501026 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:29.500599    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ll8qw" podUID="a21fd6a7-5724-4021-928b-f98404324fb8"
Jul 13 18:51:29.501026 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:29.500639    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvd96" podUID="a95bbd20-fbdc-48c9-bed2-2341f164d5e7"
Jul 13 18:51:29.646466 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.646432    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeReady"
Jul 13 18:51:29.646659 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.646587    2587 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 18:51:29.764712 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.764628    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k475c"]
Jul 13 18:51:29.766837 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.766815    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.773196 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:29.773160    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"dns-default-k475c\" is forbidden: User \"system:node:ip-10-0-139-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object" podUID="9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13" pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.773332 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:29.773285    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-dockercfg-5rh7g\" is forbidden: User \"system:node:ip-10-0-139-100.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rh7g\"" type="*v1.Secret"
Jul 13 18:51:29.773413 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.773397    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 18:51:29.773477 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:29.773395    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-default-metrics-tls\" is forbidden: User \"system:node:ip-10-0-139-100.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret"
Jul 13 18:51:29.776481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.776458    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kcvql"]
Jul 13 18:51:29.778186 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.778168    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kcvql"
Jul 13 18:51:29.781788 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.781766    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 18:51:29.781897 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.781867    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9r5lq\""
Jul 13 18:51:29.782057 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.782029    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 18:51:29.782171 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.782029    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 18:51:29.796863 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.796808    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k475c"]
Jul 13 18:51:29.819555 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.819517    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kcvql"]
Jul 13 18:51:29.874739 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.874702    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.874910 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.874759    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-config-volume\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.874910 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.874810    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2-cert\") pod \"ingress-canary-kcvql\" (UID: \"66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2\") " pod="openshift-ingress-canary/ingress-canary-kcvql"
Jul 13 18:51:29.874910 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.874835    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4nc\" (UniqueName: \"kubernetes.io/projected/66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2-kube-api-access-hc4nc\") pod \"ingress-canary-kcvql\" (UID: \"66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2\") " pod="openshift-ingress-canary/ingress-canary-kcvql"
Jul 13 18:51:29.874910 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.874861    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-tmp-dir\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.875079 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.874929    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ftmc\" (UniqueName: \"kubernetes.io/projected/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-kube-api-access-2ftmc\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.975757 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.975717    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-config-volume\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.975946 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.975766    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2-cert\") pod \"ingress-canary-kcvql\" (UID: \"66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2\") " pod="openshift-ingress-canary/ingress-canary-kcvql"
Jul 13 18:51:29.975946 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.975819    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4nc\" (UniqueName: \"kubernetes.io/projected/66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2-kube-api-access-hc4nc\") pod \"ingress-canary-kcvql\" (UID: \"66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2\") " pod="openshift-ingress-canary/ingress-canary-kcvql"
Jul 13 18:51:29.975946 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.975843    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-tmp-dir\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.975946 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.975910    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ftmc\" (UniqueName: \"kubernetes.io/projected/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-kube-api-access-2ftmc\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.976170 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.975965    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.976255 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.976235    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-tmp-dir\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.976523 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.976500    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-config-volume\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.981348 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.981096    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2-cert\") pod \"ingress-canary-kcvql\" (UID: \"66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2\") " pod="openshift-ingress-canary/ingress-canary-kcvql"
Jul 13 18:51:29.994682 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.994650    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ftmc\" (UniqueName: \"kubernetes.io/projected/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-kube-api-access-2ftmc\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:29.995153 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:29.995132    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4nc\" (UniqueName: \"kubernetes.io/projected/66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2-kube-api-access-hc4nc\") pod \"ingress-canary-kcvql\" (UID: \"66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2\") " pod="openshift-ingress-canary/ingress-canary-kcvql"
Jul 13 18:51:30.006186 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:30.006114    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:30.078192 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:30.078070    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:30.088237 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:30.088197    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kcvql"
Jul 13 18:51:30.245138 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:30.245097    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kcvql"]
Jul 13 18:51:30.248773 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:51:30.248741    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a1f85c_7204_4ef9_a3a4_fe9ae3c636e2.slice/crio-e31526cb7f8dd2c48301ab59ca904bab0d9e86e377a76024554628e601fdb763 WatchSource:0}: Error finding container e31526cb7f8dd2c48301ab59ca904bab0d9e86e377a76024554628e601fdb763: Status 404 returned error can't find the container with id e31526cb7f8dd2c48301ab59ca904bab0d9e86e377a76024554628e601fdb763
Jul 13 18:51:30.663067 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:30.663023    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kcvql" event={"ID":"66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2","Type":"ContainerStarted","Data":"e31526cb7f8dd2c48301ab59ca904bab0d9e86e377a76024554628e601fdb763"}
Jul 13 18:51:30.946625 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:30.946548    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 18:51:30.957170 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:30.957141    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:51:30.957337 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:30.957255    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls podName:9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:31.45723119 +0000 UTC m=+37.697531269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls") pod "dns-default-k475c" (UID: "9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13") : secret "dns-default-metrics-tls" not found
Jul 13 18:51:31.352466 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.352384    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rh7g\""
Jul 13 18:51:31.487671 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.487630    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:31.487844 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:31.487818    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:51:31.487898 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:31.487884    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls podName:9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:32.487867743 +0000 UTC m=+38.728167822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls") pod "dns-default-k475c" (UID: "9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13") : secret "dns-default-metrics-tls" not found
Jul 13 18:51:31.499699 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.499665    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:51:31.499867 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.499665    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:51:31.499867 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.499664    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:51:31.503130 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.503104    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:51:31.503772 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.503755    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:51:31.504147 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.504119    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 18:51:31.504147 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.504138    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2tkcn\""
Jul 13 18:51:31.504322 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.504138    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 18:51:31.504558 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.504538    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g7ddc\""
Jul 13 18:51:31.748899 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.748821    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"]
Jul 13 18:51:31.750953 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.750933    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:31.756893 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.756715    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 13 18:51:31.756893 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.756718    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 13 18:51:31.756893 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.756827    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d7b78\""
Jul 13 18:51:31.767528 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.767490    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"]
Jul 13 18:51:31.842674 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.842650    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"]
Jul 13 18:51:31.844629 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.844607    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:31.849100 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.849077    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 13 18:51:31.849264 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.849156    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 13 18:51:31.849498 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.849480    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 13 18:51:31.849980 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.849813    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:51:31.849980 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.849904    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qbpv6\""
Jul 13 18:51:31.863868 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.863839    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"]
Jul 13 18:51:31.892179 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.892145    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:31.892397 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.892301    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:31.971657 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.971568    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47"]
Jul 13 18:51:31.973814 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.973788    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c6c7c4594-qv9w2"]
Jul 13 18:51:31.974087 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.974059    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47"
Jul 13 18:51:31.976061 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.976037    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:31.983610 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.983587    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q7mrl\""
Jul 13 18:51:31.984187 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.983937    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 13 18:51:31.984187 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.983964    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 13 18:51:31.984459 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.984308    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 18:51:31.984622 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.984604    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 18:51:31.984946 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.984925    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 13 18:51:31.986164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.986122    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v94cb\""
Jul 13 18:51:31.993261 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.993233    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdjd\" (UniqueName: \"kubernetes.io/projected/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-kube-api-access-5tdjd\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:31.993365 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.993294    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:31.993365 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.993323    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-config\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:31.993462 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.993385    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-serving-cert\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:31.993508 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.993490    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:31.993876 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:31.993620    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:51:31.993876 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:31.993692    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert podName:19ecb11a-0f10-49d6-ad5e-ce2f00397c0f nodeName:}" failed. No retries permitted until 2026-07-13 18:51:32.493673828 +0000 UTC m=+38.733973921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hrpdn" (UID: "19ecb11a-0f10-49d6-ad5e-ce2f00397c0f") : secret "networking-console-plugin-cert" not found
Jul 13 18:51:31.994128 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.994082    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:31.997275 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:31.995756    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47"]
Jul 13 18:51:32.008916 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.008843    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 13 18:51:32.012674 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.012644    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c6c7c4594-qv9w2"]
Jul 13 18:51:32.047002 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.046972    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:32.094165 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094128    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.094374 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094245    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdjd\" (UniqueName: \"kubernetes.io/projected/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-kube-api-access-5tdjd\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:32.094374 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094280    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-default-certificate\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.094473 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094397    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.094473 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094428    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bxz\" (UniqueName: \"kubernetes.io/projected/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-kube-api-access-56bxz\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.094473 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094452    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmvgf\" (UniqueName: \"kubernetes.io/projected/92c24328-93b8-4f4a-abdd-dd0312ef081c-kube-api-access-cmvgf\") pod \"network-check-source-869dcfcdcd-b8h47\" (UID: \"92c24328-93b8-4f4a-abdd-dd0312ef081c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47"
Jul 13 18:51:32.094615 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094523    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-config\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:32.094615 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094596    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-stats-auth\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.094726 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.094703    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-serving-cert\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:32.095237 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.095189    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-config\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:32.098293 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.098274    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-serving-cert\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:32.103471 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.103450    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdjd\" (UniqueName: \"kubernetes.io/projected/6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68-kube-api-access-5tdjd\") pod \"service-ca-operator-7695fc6c8c-vcftq\" (UID: \"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:32.139905 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.139865    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:32.164715 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.164680    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"
Jul 13 18:51:32.195775 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.195453    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-stats-auth\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.195775 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.195724    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.195775 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.195782    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-default-certificate\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.196043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.195806    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.196043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.195831    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bxz\" (UniqueName: \"kubernetes.io/projected/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-kube-api-access-56bxz\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.196043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.195855    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmvgf\" (UniqueName: \"kubernetes.io/projected/92c24328-93b8-4f4a-abdd-dd0312ef081c-kube-api-access-cmvgf\") pod \"network-check-source-869dcfcdcd-b8h47\" (UID: \"92c24328-93b8-4f4a-abdd-dd0312ef081c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47"
Jul 13 18:51:32.196043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.195865    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:51:32.196043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.195930    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:32.695910545 +0000 UTC m=+38.936210644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : secret "router-metrics-certs-default" not found
Jul 13 18:51:32.196043 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.196012    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:32.696001358 +0000 UTC m=+38.936301444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : configmap references non-existent config key: service-ca.crt
Jul 13 18:51:32.198942 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.198901    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-default-certificate\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.199030 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.198981    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-stats-auth\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.209491 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.209464    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bxz\" (UniqueName: \"kubernetes.io/projected/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-kube-api-access-56bxz\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.210490 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.210441    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmvgf\" (UniqueName: \"kubernetes.io/projected/92c24328-93b8-4f4a-abdd-dd0312ef081c-kube-api-access-cmvgf\") pod \"network-check-source-869dcfcdcd-b8h47\" (UID: \"92c24328-93b8-4f4a-abdd-dd0312ef081c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47"
Jul 13 18:51:32.233510 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.233412    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:32.320870 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.320781    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq"]
Jul 13 18:51:32.326552 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:51:32.326520    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbc624c_d8ec_49bb_99ab_a9f9ff7b8b68.slice/crio-94e0b03c0a113a2ff335a0ca1ea5a62f0174dc86305821d896cce452f8653d2e WatchSource:0}: Error finding container 94e0b03c0a113a2ff335a0ca1ea5a62f0174dc86305821d896cce452f8653d2e: Status 404 returned error can't find the container with id 94e0b03c0a113a2ff335a0ca1ea5a62f0174dc86305821d896cce452f8653d2e
Jul 13 18:51:32.333232 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.333194    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47"
Jul 13 18:51:32.407397 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.407321    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:32.477547 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.477517    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47"]
Jul 13 18:51:32.480502 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:51:32.480463    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c24328_93b8_4f4a_abdd_dd0312ef081c.slice/crio-d690bacf8886a0c8fdd2d9a24b1f1e67de17a0a946c36e014565e83fac973d98 WatchSource:0}: Error finding container d690bacf8886a0c8fdd2d9a24b1f1e67de17a0a946c36e014565e83fac973d98: Status 404 returned error can't find the container with id d690bacf8886a0c8fdd2d9a24b1f1e67de17a0a946c36e014565e83fac973d98
Jul 13 18:51:32.481316 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.481289    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:32.498833 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.498802    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:32.498951 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.498859    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:32.499034 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.499017    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:51:32.499106 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.499094    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert podName:19ecb11a-0f10-49d6-ad5e-ce2f00397c0f nodeName:}" failed. No retries permitted until 2026-07-13 18:51:33.499073224 +0000 UTC m=+39.739373318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hrpdn" (UID: "19ecb11a-0f10-49d6-ad5e-ce2f00397c0f") : secret "networking-console-plugin-cert" not found
Jul 13 18:51:32.499303 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.499285    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:51:32.499379 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.499342    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls podName:9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:34.499327756 +0000 UTC m=+40.739627844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls") pod "dns-default-k475c" (UID: "9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13") : secret "dns-default-metrics-tls" not found
Jul 13 18:51:32.670129 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.670039    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq" event={"ID":"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68","Type":"ContainerStarted","Data":"94e0b03c0a113a2ff335a0ca1ea5a62f0174dc86305821d896cce452f8653d2e"}
Jul 13 18:51:32.673546 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.673514    2587 generic.go:356] "Generic (PLEG): container finished" podID="5bed48f8-1564-4705-9258-755e26ef91cb" containerID="b34e64a6470f2f90a70aa01d9481032ca441f208da929feabcdad20588105e08" exitCode=0
Jul 13 18:51:32.673692 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.673593    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" event={"ID":"5bed48f8-1564-4705-9258-755e26ef91cb","Type":"ContainerDied","Data":"b34e64a6470f2f90a70aa01d9481032ca441f208da929feabcdad20588105e08"}
Jul 13 18:51:32.674865 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.674841    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47" event={"ID":"92c24328-93b8-4f4a-abdd-dd0312ef081c","Type":"ContainerStarted","Data":"d690bacf8886a0c8fdd2d9a24b1f1e67de17a0a946c36e014565e83fac973d98"}
Jul 13 18:51:32.700249 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.700196    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.700364 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:32.700296    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:32.700364 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.700314    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:51:32.700463 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.700368    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:33.700352727 +0000 UTC m=+39.940652804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : secret "router-metrics-certs-default" not found
Jul 13 18:51:32.700463 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:32.700445    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:33.700426261 +0000 UTC m=+39.940726356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : configmap references non-existent config key: service-ca.crt
Jul 13 18:51:33.507383 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:33.507337    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:33.508177 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:33.507506    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:51:33.508177 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:33.507589    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert podName:19ecb11a-0f10-49d6-ad5e-ce2f00397c0f nodeName:}" failed. No retries permitted until 2026-07-13 18:51:35.507569087 +0000 UTC m=+41.747869183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hrpdn" (UID: "19ecb11a-0f10-49d6-ad5e-ce2f00397c0f") : secret "networking-console-plugin-cert" not found
Jul 13 18:51:33.680080 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:33.679998    2587 generic.go:356] "Generic (PLEG): container finished" podID="5bed48f8-1564-4705-9258-755e26ef91cb" containerID="d549d14b2486b8ec1e608226f6c9835324aadd28ec0ce51a63c19de16325d3f9" exitCode=0
Jul 13 18:51:33.680080 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:33.680061    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" event={"ID":"5bed48f8-1564-4705-9258-755e26ef91cb","Type":"ContainerDied","Data":"d549d14b2486b8ec1e608226f6c9835324aadd28ec0ce51a63c19de16325d3f9"}
Jul 13 18:51:33.709625 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:33.709599    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:33.709766 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:33.709646    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:33.709766 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:33.709750    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:35.709736746 +0000 UTC m=+41.950036824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : configmap references non-existent config key: service-ca.crt
Jul 13 18:51:33.709766 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:33.709755    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:51:33.709867 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:33.709807    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:35.709793099 +0000 UTC m=+41.950093182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : secret "router-metrics-certs-default" not found
Jul 13 18:51:34.518238 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:34.517982    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:34.518825 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:34.518128    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:51:34.518825 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:34.518445    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls podName:9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:38.518423634 +0000 UTC m=+44.758723730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls") pod "dns-default-k475c" (UID: "9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13") : secret "dns-default-metrics-tls" not found
Jul 13 18:51:34.687285 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:34.687244    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" event={"ID":"5bed48f8-1564-4705-9258-755e26ef91cb","Type":"ContainerStarted","Data":"300de5b54d94fa11023a66a7b63f0211321f9693c9b2f4355dfc55d3fdc261bc"}
Jul 13 18:51:34.688902 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:34.688873    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kcvql" event={"ID":"66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2","Type":"ContainerStarted","Data":"7fbc61aafc3118ef437fbc5e0a20759113e55a398aee39e842bdef618efd6d11"}
Jul 13 18:51:34.704860 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:34.704821    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:34.716526 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:34.716479    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dhgp5" podStartSLOduration=9.242658214 podStartE2EDuration="40.71646457s" totalImagesPullingTime="31.473806356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:34.714976496 +0000 UTC m=+40.955276599" watchObservedRunningTime="2026-07-13 18:51:34.71646457 +0000 UTC m=+40.956764671"
Jul 13 18:51:34.731425 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:34.731317    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kcvql" podStartSLOduration=2.261136944 podStartE2EDuration="5.731298881s" totalImagesPullingTime="3.470161937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:34.73112099 +0000 UTC m=+40.971421089" watchObservedRunningTime="2026-07-13 18:51:34.731298881 +0000 UTC m=+40.971598980"
Jul 13 18:51:34.802046 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:34.801948    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:34.902818 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:34.902771    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:35.527472 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:35.527439    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:35.527824 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:35.527595    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:51:35.527824 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:35.527676    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert podName:19ecb11a-0f10-49d6-ad5e-ce2f00397c0f nodeName:}" failed. No retries permitted until 2026-07-13 18:51:39.527658562 +0000 UTC m=+45.767958639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hrpdn" (UID: "19ecb11a-0f10-49d6-ad5e-ce2f00397c0f") : secret "networking-console-plugin-cert" not found
Jul 13 18:51:35.729384 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:35.729348    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:35.729580 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:35.729510    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:39.729485374 +0000 UTC m=+45.969785464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : configmap references non-existent config key: service-ca.crt
Jul 13 18:51:35.729716 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:35.729688    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:35.729841 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:35.729799    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:51:35.729899 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:35.729861    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:39.729843746 +0000 UTC m=+45.970143825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : secret "router-metrics-certs-default" not found
Jul 13 18:51:35.981073 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:35.980994    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x8zfs"]
Jul 13 18:51:35.998143 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:35.998112    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.000866 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.000834    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x8zfs"]
Jul 13 18:51:36.001107 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.001021    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 18:51:36.001107 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.001097    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9bc6\""
Jul 13 18:51:36.001421 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.001366    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 18:51:36.001726 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.001579    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 18:51:36.002338 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.002311    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 18:51:36.031017 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.030981    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7de805b-a750-4359-afc3-61b3e1d2cac7-crio-socket\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.031170 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.031030    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knk6s\" (UniqueName: \"kubernetes.io/projected/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-api-access-knk6s\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.031170 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.031103    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.031170 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.031125    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.031170 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.031146    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7de805b-a750-4359-afc3-61b3e1d2cac7-data-volume\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.132378 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.132341    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7de805b-a750-4359-afc3-61b3e1d2cac7-crio-socket\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.132560 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.132395    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knk6s\" (UniqueName: \"kubernetes.io/projected/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-api-access-knk6s\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.132560 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.132455    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.132560 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.132481    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.132560 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.132507    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7de805b-a750-4359-afc3-61b3e1d2cac7-data-volume\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.132767 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:36.132582    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:36.132767 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:36.132655    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:36.132767 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:36.132666    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:36.63264192 +0000 UTC m=+42.872942002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:36.132767 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:36.132712    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:36.632696598 +0000 UTC m=+42.872996676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:36.132767 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.132590    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7de805b-a750-4359-afc3-61b3e1d2cac7-crio-socket\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.133019 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.132844    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7de805b-a750-4359-afc3-61b3e1d2cac7-data-volume\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.147696 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.147657    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knk6s\" (UniqueName: \"kubernetes.io/projected/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-api-access-knk6s\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.254615 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.254514    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:36.362854 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.362812    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:36.636518 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.636443    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.636518 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.636483    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:36.636889 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:36.636574    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:36.636889 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:36.636627    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:36.636889 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:36.636646    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:37.63662709 +0000 UTC m=+43.876927183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:36.636889 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:36.636667    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:37.636657036 +0000 UTC m=+43.876957113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:36.838006 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.837970    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:36.866070 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.866043    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j"]
Jul 13 18:51:36.887145 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.887086    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j"
Jul 13 18:51:36.890392 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.890368    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 13 18:51:36.890392 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.890379    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 13 18:51:36.891620 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.891601    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5mhn9\""
Jul 13 18:51:36.919505 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.897635    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j"]
Jul 13 18:51:36.938242 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.938196    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4fl\" (UniqueName: \"kubernetes.io/projected/2dd5bac6-7970-444d-8238-328f4af61ab2-kube-api-access-rg4fl\") pod \"migrator-7f59b54f6c-5q57j\" (UID: \"2dd5bac6-7970-444d-8238-328f4af61ab2\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j"
Jul 13 18:51:36.938377 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:36.938357    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:37.035891 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.035860    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:37.039505 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.039482    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4fl\" (UniqueName: \"kubernetes.io/projected/2dd5bac6-7970-444d-8238-328f4af61ab2-kube-api-access-rg4fl\") pod \"migrator-7f59b54f6c-5q57j\" (UID: \"2dd5bac6-7970-444d-8238-328f4af61ab2\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j"
Jul 13 18:51:37.048841 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.048816    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4fl\" (UniqueName: \"kubernetes.io/projected/2dd5bac6-7970-444d-8238-328f4af61ab2-kube-api-access-rg4fl\") pod \"migrator-7f59b54f6c-5q57j\" (UID: \"2dd5bac6-7970-444d-8238-328f4af61ab2\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j"
Jul 13 18:51:37.196733 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.196622    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j"
Jul 13 18:51:37.347616 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.347584    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j"]
Jul 13 18:51:37.350432 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:51:37.350406    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd5bac6_7970_444d_8238_328f4af61ab2.slice/crio-290b5d81ba416f3aca0c6283ea3595ea1b927a998343d2f6e183c0058dfeb3d7 WatchSource:0}: Error finding container 290b5d81ba416f3aca0c6283ea3595ea1b927a998343d2f6e183c0058dfeb3d7: Status 404 returned error can't find the container with id 290b5d81ba416f3aca0c6283ea3595ea1b927a998343d2f6e183c0058dfeb3d7
Jul 13 18:51:37.644995 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.644962    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:37.645450 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:37.645118    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:37.645450 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.645128    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:37.645450 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:37.645196    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:39.645174797 +0000 UTC m=+45.885474886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:37.645450 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:37.645225    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:37.645450 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:37.645266    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:39.645257398 +0000 UTC m=+45.885557476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:37.698839 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.698801    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47" event={"ID":"92c24328-93b8-4f4a-abdd-dd0312ef081c","Type":"ContainerStarted","Data":"e664c1f043989aebafc5d51559a7e53672b8ffd91d5661326e5fe82eb73f4647"}
Jul 13 18:51:37.700421 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.700393    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq" event={"ID":"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68","Type":"ContainerStarted","Data":"f41c8c196255d27f0bc55c1dbc24cb81829538f2848d6f40f5fc335e933c5dc6"}
Jul 13 18:51:37.701595 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.701573    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j" event={"ID":"2dd5bac6-7970-444d-8238-328f4af61ab2","Type":"ContainerStarted","Data":"290b5d81ba416f3aca0c6283ea3595ea1b927a998343d2f6e183c0058dfeb3d7"}
Jul 13 18:51:37.716244 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.716173    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-b8h47" podStartSLOduration=2.270854762 podStartE2EDuration="6.716159019s" totalImagesPullingTime="4.445304257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:37.714383054 +0000 UTC m=+43.954683152" watchObservedRunningTime="2026-07-13 18:51:37.716159019 +0000 UTC m=+43.956459165"
Jul 13 18:51:37.731415 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.731368    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq" podStartSLOduration=2.138754303 podStartE2EDuration="6.731356566s" totalImagesPullingTime="4.592602263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:37.730086624 +0000 UTC m=+43.970386724" watchObservedRunningTime="2026-07-13 18:51:37.731356566 +0000 UTC m=+43.971656667"
Jul 13 18:51:37.949424 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:37.949345    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqc2j_4b227090-d946-4bf0-be0f-cba5485fd90f/dns-node-resolver/0.log"
Jul 13 18:51:38.554132 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:38.554095    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:38.554324 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:38.554282    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:51:38.554397 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:38.554357    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls podName:9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:46.55433826 +0000 UTC m=+52.794638337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls") pod "dns-default-k475c" (UID: "9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13") : secret "dns-default-metrics-tls" not found
Jul 13 18:51:38.650781 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:38.650745    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:38.716978 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:38.716943    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:38.949555 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:38.949490    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8fl4m_29c31923-0274-4781-be82-8266a0459019/node-ca/0.log"
Jul 13 18:51:39.356495 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.356463    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:39.448036 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.448002    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:39.562597 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.562567    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:39.562795 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.562749    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:51:39.562863 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.562839    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert podName:19ecb11a-0f10-49d6-ad5e-ce2f00397c0f nodeName:}" failed. No retries permitted until 2026-07-13 18:51:47.56281628 +0000 UTC m=+53.803116390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hrpdn" (UID: "19ecb11a-0f10-49d6-ad5e-ce2f00397c0f") : secret "networking-console-plugin-cert" not found
Jul 13 18:51:39.667103 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.667067    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:39.667103 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.667105    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:39.667663 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.667230    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:39.667663 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.667261    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:39.667663 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.667326    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:43.667304038 +0000 UTC m=+49.907604134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:39.667663 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.667346    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:43.667337163 +0000 UTC m=+49.907637241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:39.708315 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.708276    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j" event={"ID":"2dd5bac6-7970-444d-8238-328f4af61ab2","Type":"ContainerStarted","Data":"96b0a2298aedc7f41442d0235e3f52187c031a7f7c346f0a3380716bf658b7de"}
Jul 13 18:51:39.708493 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.708356    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j" event={"ID":"2dd5bac6-7970-444d-8238-328f4af61ab2","Type":"ContainerStarted","Data":"07f15b90aa809c08338b463b10e89e8b7901cf99126f5c9d20511072c1c580cb"}
Jul 13 18:51:39.734665 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.734618    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-5q57j" podStartSLOduration=1.987113347 podStartE2EDuration="3.734603032s" totalImagesPullingTime="1.747489685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:39.734050758 +0000 UTC m=+45.974350857" watchObservedRunningTime="2026-07-13 18:51:39.734603032 +0000 UTC m=+45.974903131"
Jul 13 18:51:39.767621 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.767589    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:39.767763 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.767644    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:39.767800 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.767770    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:51:39.767833 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.767820    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:47.767801987 +0000 UTC m=+54.008102065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : configmap references non-existent config key: service-ca.crt
Jul 13 18:51:39.767871 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:39.767835    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:47.767828643 +0000 UTC m=+54.008128721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : secret "router-metrics-certs-default" not found
Jul 13 18:51:39.838451 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.838382    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:39.940800 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.940764    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:39.953116 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:39.953085    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kcvql_66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2/serve-healthcheck-canary/0.log"
Jul 13 18:51:41.003800 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:41.003761    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:41.067987 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:41.067954    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:42.046834 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:42.046799    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:43.698240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:43.698194    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:43.698240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:43.698244    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:43.698724 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:43.698403    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:43.698724 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:43.698476    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:51.698459649 +0000 UTC m=+57.938759727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:43.698724 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:43.698410    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:43.698724 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:43.698557    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:51.698538798 +0000 UTC m=+57.938838876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:45.524910 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:45.524868    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:46.621702 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:46.621659    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:51:46.622105 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:46.621822    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:51:46.622105 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:46.621888    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls podName:9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:02.621873033 +0000 UTC m=+68.862173110 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls") pod "dns-default-k475c" (UID: "9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13") : secret "dns-default-metrics-tls" not found
Jul 13 18:51:47.337927 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:47.337892    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:47.437861 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:47.437829    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:47.630217 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:47.630100    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:51:47.630667 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:47.630294    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:51:47.630667 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:47.630381    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert podName:19ecb11a-0f10-49d6-ad5e-ce2f00397c0f nodeName:}" failed. No retries permitted until 2026-07-13 18:52:03.630359059 +0000 UTC m=+69.870659159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hrpdn" (UID: "19ecb11a-0f10-49d6-ad5e-ce2f00397c0f") : secret "networking-console-plugin-cert" not found
Jul 13 18:51:47.736477 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:47.736439    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:47.831416 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:47.831374    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:47.831586 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:47.831433    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:51:47.831586 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:47.831517    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:51:47.831586 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:47.831565    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:03.831551161 +0000 UTC m=+70.071851238 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : configmap references non-existent config key: service-ca.crt
Jul 13 18:51:47.831586 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:47.831580    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs podName:8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:03.831573669 +0000 UTC m=+70.071873746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs") pod "router-default-6c6c7c4594-qv9w2" (UID: "8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7") : secret "router-metrics-certs-default" not found
Jul 13 18:51:47.836949 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:47.836922    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:48.056223 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:48.056169    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:48.639224 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:48.639174    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:48.955406 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:48.955310    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:49.448328 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:49.448289    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:49.651512 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:49.651474    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:50.055145 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:50.055112    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:50.249476 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:50.249431    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:50.642226 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:50.642178    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:50.949219 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:50.949083    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:51.050123 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:51.050088    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:51.448136 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:51.448106    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:51.763988 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:51.763907    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:51.763988 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:51.763945    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:51.764165 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:51.764086    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:51.764165 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:51:51.764159    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls podName:b7de805b-a750-4359-afc3-61b3e1d2cac7 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:07.764139386 +0000 UTC m=+74.004439465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x8zfs" (UID: "b7de805b-a750-4359-afc3-61b3e1d2cac7") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:51.764614 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:51.764594    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7de805b-a750-4359-afc3-61b3e1d2cac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:51:51.834727 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:51.834678    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:51.935925 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:51.935891    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:52.042613 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:52.042577    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:52.240280 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:52.240242    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:52.666591 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:52.666552    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:53.443779 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:53.443744    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:53.728717 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:53.728636    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:53.798707 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:53.798665    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:53.881514 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:53.881483    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:54.042258 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:54.042185    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:54.244862 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:54.244828    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:54.446597 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:54.446507    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:55.637944 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:55.637908    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:55.666401 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:55.666372    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fsmf"
Jul 13 18:51:55.837493 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:55.837454    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:56.033384 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:56.033343    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:56.241810 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:56.241774    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:56.434174 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:56.434087    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:57.232677 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:57.232636    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:57.429866 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:57.429836    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:57.679410 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:57.679361    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:58.037021 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:51:58.036987    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:00.131262 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.131188    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:52:00.134273 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.134255    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:52:00.143705 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.143684    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7146a5f-6f9b-417d-94f3-8aabf7e61501-original-pull-secret\") pod \"global-pull-secret-syncer-8gk2r\" (UID: \"d7146a5f-6f9b-417d-94f3-8aabf7e61501\") " pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:52:00.232344 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.232306    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:52:00.235572 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.235553    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:52:00.245587 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.245558    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21fd6a7-5724-4021-928b-f98404324fb8-metrics-certs\") pod \"network-metrics-daemon-ll8qw\" (UID: \"a21fd6a7-5724-4021-928b-f98404324fb8\") " pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:52:00.312104 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.312054    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gk2r"
Jul 13 18:52:00.323239 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.323216    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2tkcn\""
Jul 13 18:52:00.331012 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.330989    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ll8qw"
Jul 13 18:52:00.332787 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.332759    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:52:00.335404 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.335373    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgcjz\" (UniqueName: \"kubernetes.io/projected/a95bbd20-fbdc-48c9-bed2-2341f164d5e7-kube-api-access-jgcjz\") pod \"network-check-target-xvd96\" (UID: \"a95bbd20-fbdc-48c9-bed2-2341f164d5e7\") " pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:52:00.443654 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.443621    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8gk2r"]
Jul 13 18:52:00.446972 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:00.446939    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7146a5f_6f9b_417d_94f3_8aabf7e61501.slice/crio-40c10f0e75ffa8fe2fde1c0a2bdcc499002d0c747496eff10f28800fc26be610 WatchSource:0}: Error finding container 40c10f0e75ffa8fe2fde1c0a2bdcc499002d0c747496eff10f28800fc26be610: Status 404 returned error can't find the container with id 40c10f0e75ffa8fe2fde1c0a2bdcc499002d0c747496eff10f28800fc26be610
Jul 13 18:52:00.462287 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.462256    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ll8qw"]
Jul 13 18:52:00.465749 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:00.465719    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21fd6a7_5724_4021_928b_f98404324fb8.slice/crio-ae6b4670443542b67f0eac1aef06fb404ea5ef18ce0e93b050ba6fd397a418cb WatchSource:0}: Error finding container ae6b4670443542b67f0eac1aef06fb404ea5ef18ce0e93b050ba6fd397a418cb: Status 404 returned error can't find the container with id ae6b4670443542b67f0eac1aef06fb404ea5ef18ce0e93b050ba6fd397a418cb
Jul 13 18:52:00.629608 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.629574    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g7ddc\""
Jul 13 18:52:00.637123 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.637065    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:52:00.756779 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.756753    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xvd96"]
Jul 13 18:52:00.759369 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:00.759340    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95bbd20_fbdc_48c9_bed2_2341f164d5e7.slice/crio-0635d45a6733cd1ef8f68196ae218e21213fb89582051ed54191ffb29c3cf3eb WatchSource:0}: Error finding container 0635d45a6733cd1ef8f68196ae218e21213fb89582051ed54191ffb29c3cf3eb: Status 404 returned error can't find the container with id 0635d45a6733cd1ef8f68196ae218e21213fb89582051ed54191ffb29c3cf3eb
Jul 13 18:52:00.770862 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.770832    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xvd96" event={"ID":"a95bbd20-fbdc-48c9-bed2-2341f164d5e7","Type":"ContainerStarted","Data":"0635d45a6733cd1ef8f68196ae218e21213fb89582051ed54191ffb29c3cf3eb"}
Jul 13 18:52:00.771908 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.771885    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ll8qw" event={"ID":"a21fd6a7-5724-4021-928b-f98404324fb8","Type":"ContainerStarted","Data":"ae6b4670443542b67f0eac1aef06fb404ea5ef18ce0e93b050ba6fd397a418cb"}
Jul 13 18:52:00.772951 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:00.772926    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8gk2r" event={"ID":"d7146a5f-6f9b-417d-94f3-8aabf7e61501","Type":"ContainerStarted","Data":"40c10f0e75ffa8fe2fde1c0a2bdcc499002d0c747496eff10f28800fc26be610"}
Jul 13 18:52:01.778614 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.778578    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xvd96" event={"ID":"a95bbd20-fbdc-48c9-bed2-2341f164d5e7","Type":"ContainerStarted","Data":"d4c5d72294f399625f5835674a9d3bb0f3c6a56b17d2b4bbd7910f8ba752e97c"}
Jul 13 18:52:01.779056 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.778685    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:52:01.780896 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.780832    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ll8qw" event={"ID":"a21fd6a7-5724-4021-928b-f98404324fb8","Type":"ContainerStarted","Data":"1d15590969e3e93ac9ac6eb9f4a121279452c4e78ed4e127bfca87c0cd5927f7"}
Jul 13 18:52:01.780896 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.780872    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ll8qw" event={"ID":"a21fd6a7-5724-4021-928b-f98404324fb8","Type":"ContainerStarted","Data":"67c7955d988f690eb604d8f9db5df48e4ce8b41d5697c369907b5f452ef6f352"}
Jul 13 18:52:01.799313 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.799273    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"]
Jul 13 18:52:01.801862 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.801840    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:01.804765 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.804705    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 13 18:52:01.805953 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.805930    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 13 18:52:01.806072 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.805983    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 13 18:52:01.806164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.806143    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 13 18:52:01.806295 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.806248    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 13 18:52:01.806295 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.806259    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 13 18:52:01.806409 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.806345    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 13 18:52:01.808343 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.808300    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xvd96" podStartSLOduration=67.808284186 podStartE2EDuration="1m7.808284186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:01.807241051 +0000 UTC m=+68.047541155" watchObservedRunningTime="2026-07-13 18:52:01.808284186 +0000 UTC m=+68.048584286"
Jul 13 18:52:01.819841 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.819814    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"]
Jul 13 18:52:01.860161 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.860112    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ll8qw" podStartSLOduration=66.880331964 podStartE2EDuration="1m7.860097917s" totalImagesPullingTime="979.765953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:01.858999985 +0000 UTC m=+68.099300086" watchObservedRunningTime="2026-07-13 18:52:01.860097917 +0000 UTC m=+68.100398017"
Jul 13 18:52:01.946838 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.946797    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j756r\" (UniqueName: \"kubernetes.io/projected/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-kube-api-access-j756r\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:01.947020 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.946881    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:01.947020 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.946911    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-hub\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:01.947020 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.946939    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:01.947020 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.946984    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:01.947222 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:01.947126    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-ca\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.001381 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.001347    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:02.047710 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.047627    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j756r\" (UniqueName: \"kubernetes.io/projected/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-kube-api-access-j756r\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.047710 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.047681    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.047710 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.047700    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-hub\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.047959 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.047717    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.047959 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.047756    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.047959 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.047818    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-ca\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.048578 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.048552    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.050381 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.050358    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-ca\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.050505 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.050480    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.050609 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.050592    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.050691 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.050674    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-hub\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.060507 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.060487    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j756r\" (UniqueName: \"kubernetes.io/projected/a095daae-c1c3-4ea1-bc35-ec0a1514e3eb-kube-api-access-j756r\") pod \"cluster-proxy-proxy-agent-86d986d476-4x6dn\" (UID: \"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.124984 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.124950    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"
Jul 13 18:52:02.134716 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.134628    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:02.227630 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.227572    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:02.284553 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.284486    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn"]
Jul 13 18:52:02.288456 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:02.288431    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda095daae_c1c3_4ea1_bc35_ec0a1514e3eb.slice/crio-5756dfa4f1d507601d24e820598caa9dade3bec32fcb30ea9f9a090019a9135f WatchSource:0}: Error finding container 5756dfa4f1d507601d24e820598caa9dade3bec32fcb30ea9f9a090019a9135f: Status 404 returned error can't find the container with id 5756dfa4f1d507601d24e820598caa9dade3bec32fcb30ea9f9a090019a9135f
Jul 13 18:52:02.353838 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.353761    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:02.455382 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.455346    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:02.653342 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.653265    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:52:02.656012 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.655984    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13-metrics-tls\") pod \"dns-default-k475c\" (UID: \"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13\") " pod="openshift-dns/dns-default-k475c"
Jul 13 18:52:02.781667 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.781633    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rh7g\""
Jul 13 18:52:02.784859 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.784829    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn" event={"ID":"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb","Type":"ContainerStarted","Data":"5756dfa4f1d507601d24e820598caa9dade3bec32fcb30ea9f9a090019a9135f"}
Jul 13 18:52:02.789364 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:02.789342    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k475c"
Jul 13 18:52:03.660403 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:03.660365    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:52:03.662886 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:03.662854    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19ecb11a-0f10-49d6-ad5e-ce2f00397c0f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hrpdn\" (UID: \"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:52:03.862049 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:03.862007    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:03.862521 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:03.862068    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:03.862521 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:03.862166    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d7b78\""
Jul 13 18:52:03.862846 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:03.862818    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-service-ca-bundle\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:03.864779 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:03.864752    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7-metrics-certs\") pod \"router-default-6c6c7c4594-qv9w2\" (UID: \"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7\") " pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:03.870590 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:03.870561    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"
Jul 13 18:52:04.141978 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.141947    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q7mrl\""
Jul 13 18:52:04.148681 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.148646    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:04.393771 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.393683    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c6c7c4594-qv9w2"]
Jul 13 18:52:04.397592 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:04.397559    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec3b9d5_cd39_418d_8c52_d8cdc7bd10e7.slice/crio-e3d628ad20b2d10b5f53b5c58aa56d56d4ca29c9d9504fc4a0ddcf2e4ff09f94 WatchSource:0}: Error finding container e3d628ad20b2d10b5f53b5c58aa56d56d4ca29c9d9504fc4a0ddcf2e4ff09f94: Status 404 returned error can't find the container with id e3d628ad20b2d10b5f53b5c58aa56d56d4ca29c9d9504fc4a0ddcf2e4ff09f94
Jul 13 18:52:04.406597 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.406469    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-hrpdn"]
Jul 13 18:52:04.423815 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.423715    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k475c"]
Jul 13 18:52:04.580528 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:04.580491    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ecb11a_0f10_49d6_ad5e_ce2f00397c0f.slice/crio-8d085405df689bfcdd74d61fa0bbf58ceaae4db9bcde60d9a27e34e081352dce WatchSource:0}: Error finding container 8d085405df689bfcdd74d61fa0bbf58ceaae4db9bcde60d9a27e34e081352dce: Status 404 returned error can't find the container with id 8d085405df689bfcdd74d61fa0bbf58ceaae4db9bcde60d9a27e34e081352dce
Jul 13 18:52:04.581065 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:04.581025    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cfe0588_1fa2_4cc4_a5b3_3eeaae8abb13.slice/crio-44a8ee7c633ff88b02e0013603fc5a8e0401b0d4fb1a367e3ba7559f8641b38c WatchSource:0}: Error finding container 44a8ee7c633ff88b02e0013603fc5a8e0401b0d4fb1a367e3ba7559f8641b38c: Status 404 returned error can't find the container with id 44a8ee7c633ff88b02e0013603fc5a8e0401b0d4fb1a367e3ba7559f8641b38c
Jul 13 18:52:04.795273 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.794504    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8gk2r" event={"ID":"d7146a5f-6f9b-417d-94f3-8aabf7e61501","Type":"ContainerStarted","Data":"461464b07bd74215b3b3969effb534ebfc1f20ac418b129ca9e18fb2a3a115ad"}
Jul 13 18:52:04.797481 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.797417    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn" event={"ID":"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f","Type":"ContainerStarted","Data":"8d085405df689bfcdd74d61fa0bbf58ceaae4db9bcde60d9a27e34e081352dce"}
Jul 13 18:52:04.798961 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.798873    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k475c" event={"ID":"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13","Type":"ContainerStarted","Data":"44a8ee7c633ff88b02e0013603fc5a8e0401b0d4fb1a367e3ba7559f8641b38c"}
Jul 13 18:52:04.800730 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.800676    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2" event={"ID":"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7","Type":"ContainerStarted","Data":"fc1aea6e02fbd498457a861f93bb2dd4f522d8577d171ef812bf803da1564e8c"}
Jul 13 18:52:04.800730 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.800716    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2" event={"ID":"8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7","Type":"ContainerStarted","Data":"e3d628ad20b2d10b5f53b5c58aa56d56d4ca29c9d9504fc4a0ddcf2e4ff09f94"}
Jul 13 18:52:04.834470 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.834409    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2" podStartSLOduration=33.83438905 podStartE2EDuration="33.83438905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:04.834229732 +0000 UTC m=+71.074529835" watchObservedRunningTime="2026-07-13 18:52:04.83438905 +0000 UTC m=+71.074689154"
Jul 13 18:52:04.835326 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:04.835287    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8gk2r" podStartSLOduration=66.665197682 podStartE2EDuration="1m10.835270471s" totalImagesPullingTime="4.170072789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:04.813227987 +0000 UTC m=+71.053528087" watchObservedRunningTime="2026-07-13 18:52:04.835270471 +0000 UTC m=+71.075570574"
Jul 13 18:52:05.149413 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:05.149333    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:05.152408 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:05.152383    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:05.242454 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:05.242412    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:05.308565 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:05.308530    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:05.807271 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:05.807233    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:05.808650 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:05.808622    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c6c7c4594-qv9w2"
Jul 13 18:52:07.796185 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:07.796144    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:52:07.798746 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:07.798714    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7de805b-a750-4359-afc3-61b3e1d2cac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x8zfs\" (UID: \"b7de805b-a750-4359-afc3-61b3e1d2cac7\") " pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:52:07.812345 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:07.812317    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9bc6\""
Jul 13 18:52:07.820649 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:07.820624    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x8zfs"
Jul 13 18:52:08.447865 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.447740    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x8zfs"]
Jul 13 18:52:08.453810 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:08.453751    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7de805b_a750_4359_afc3_61b3e1d2cac7.slice/crio-e3ba31cd6fdd7a49a5786056fd03414e3d5f46a2a1af97695549f402f6520b41 WatchSource:0}: Error finding container e3ba31cd6fdd7a49a5786056fd03414e3d5f46a2a1af97695549f402f6520b41: Status 404 returned error can't find the container with id e3ba31cd6fdd7a49a5786056fd03414e3d5f46a2a1af97695549f402f6520b41
Jul 13 18:52:08.818735 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.818690    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x8zfs" event={"ID":"b7de805b-a750-4359-afc3-61b3e1d2cac7","Type":"ContainerStarted","Data":"e3ba31cd6fdd7a49a5786056fd03414e3d5f46a2a1af97695549f402f6520b41"}
Jul 13 18:52:08.820990 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.820963    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn" event={"ID":"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb","Type":"ContainerStarted","Data":"f8d11b3d385211326aae4f7341c29febb8fbf9a1390498f8504e3f43c99c012f"}
Jul 13 18:52:08.821124 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.821005    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn" event={"ID":"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb","Type":"ContainerStarted","Data":"a5493f35f6d1386a82c7c40b4e94d5bd6bb609424af94f16d0eb80092b62f28f"}
Jul 13 18:52:08.821124 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.821024    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn" event={"ID":"a095daae-c1c3-4ea1-bc35-ec0a1514e3eb","Type":"ContainerStarted","Data":"186c0fefcf9768659d9721813bce77c48d35192d67cf596555c0a84471f8ca4e"}
Jul 13 18:52:08.823016 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.822992    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k475c" event={"ID":"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13","Type":"ContainerStarted","Data":"c81bf236fee972ce727dba50a96cd4cc532e50a7ef9dd9f8062fcf3ad827973a"}
Jul 13 18:52:08.823137 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.823041    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k475c" event={"ID":"9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13","Type":"ContainerStarted","Data":"2250a58f0f33485c9e5556997f80a9f77a5aff9246847b0612c2e086447ca00d"}
Jul 13 18:52:08.823185 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.823148    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k475c"
Jul 13 18:52:08.841227 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.841116    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d986d476-4x6dn" podStartSLOduration=1.816447833 podStartE2EDuration="7.841092822s" totalImagesPullingTime="6.024644989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:08.840868942 +0000 UTC m=+75.081169055" watchObservedRunningTime="2026-07-13 18:52:08.841092822 +0000 UTC m=+75.081392922"
Jul 13 18:52:08.861140 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:08.861082    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k475c" podStartSLOduration=36.158839324 podStartE2EDuration="39.861061784s" totalImagesPullingTime="3.70222246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:08.859676099 +0000 UTC m=+75.099976206" watchObservedRunningTime="2026-07-13 18:52:08.861061784 +0000 UTC m=+75.101361936"
Jul 13 18:52:09.828143 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:09.828101    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn" event={"ID":"19ecb11a-0f10-49d6-ad5e-ce2f00397c0f","Type":"ContainerStarted","Data":"3489948b913bfdb591f4457a2e0ccd6ce431b9c041b4538a3d0f75cc8fd80fbf"}
Jul 13 18:52:09.847039 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:09.846990    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-hrpdn" podStartSLOduration=34.393029879 podStartE2EDuration="38.84697301s" totalImagesPullingTime="4.453943131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:09.84636531 +0000 UTC m=+76.086665413" watchObservedRunningTime="2026-07-13 18:52:09.84697301 +0000 UTC m=+76.087273110"
Jul 13 18:52:10.089246 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:10.089113    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:10.155102 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:10.155056    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:12.842539 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:12.842503    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x8zfs" event={"ID":"b7de805b-a750-4359-afc3-61b3e1d2cac7","Type":"ContainerStarted","Data":"6fe23fcea669a6df5be21082263c53e90f55d61b43ad9fa9511a6e84b686d1b5"}
Jul 13 18:52:13.848317 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:13.848278    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x8zfs" event={"ID":"b7de805b-a750-4359-afc3-61b3e1d2cac7","Type":"ContainerStarted","Data":"778fd8726b835479a6cea1310903d3fd0b002e2057d8ad8967d22ec84742e01a"}
Jul 13 18:52:14.607296 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:14.607251    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:14.673287 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:14.673236    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:14.859457 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:14.859366    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:15.857736 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:15.857599    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x8zfs" event={"ID":"b7de805b-a750-4359-afc3-61b3e1d2cac7","Type":"ContainerStarted","Data":"f2b634f67dd30891f94abcff7d8ba1ed21c9c8562661f0800bf6bbcdae208e44"}
Jul 13 18:52:15.879334 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:15.879280    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x8zfs" podStartSLOduration=34.05389157 podStartE2EDuration="40.879263895s" totalImagesPullingTime="6.825372325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:51:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:15.877456121 +0000 UTC m=+82.117756234" watchObservedRunningTime="2026-07-13 18:52:15.879263895 +0000 UTC m=+82.119563995"
Jul 13 18:52:18.830139 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:18.830105    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k475c"
Jul 13 18:52:20.190590 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:20.190558    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:22.796040 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:22.796002    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:22.992408 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:22.992369    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:23.918058 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:23.918024    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:24.020787 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:24.020751    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:24.329343 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:24.329291    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:24.405570 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:24.405531    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:25.567995 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:25.567962    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:25.642723 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:25.642690    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:26.771534 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:26.771495    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:26.910614 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:26.910560    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:26.983178 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:26.983136    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j59w7"]
Jul 13 18:52:26.996709 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:26.996669    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:26.999547 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:26.999523    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 18:52:26.999707 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:26.999572    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6pnlk\""
Jul 13 18:52:27.000123 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.000102    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 18:52:27.000293 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.000233    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 18:52:27.000807 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.000791    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 18:52:27.000883 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.000818    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 18:52:27.000940 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.000900    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 18:52:27.015271 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.015240    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:27.102779 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.102689    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:27.144944 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.144909    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-sys\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.145091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.144960    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-textfile\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.145091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.144987    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-root\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.145091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.145010    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-tls\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.145091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.145033    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9sk4\" (UniqueName: \"kubernetes.io/projected/946481d5-3192-42a0-949d-6494bc388aa6-kube-api-access-r9sk4\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.145091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.145053    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/946481d5-3192-42a0-949d-6494bc388aa6-metrics-client-ca\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.145091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.145073    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-wtmp\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.145324 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.145097    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.145324 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.145146    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.204916 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.204884    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:27.246467 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246434    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-textfile\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246467 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246465    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-root\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246669 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246484    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-tls\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246669 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246557    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-root\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246669 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246615    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9sk4\" (UniqueName: \"kubernetes.io/projected/946481d5-3192-42a0-949d-6494bc388aa6-kube-api-access-r9sk4\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246669 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246659    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/946481d5-3192-42a0-949d-6494bc388aa6-metrics-client-ca\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246818 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246697    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-wtmp\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246818 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246733    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246818 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246767    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246958 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246835    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-sys\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246958 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246847    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-textfile\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246958 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246884    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-wtmp\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.246958 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.246901    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/946481d5-3192-42a0-949d-6494bc388aa6-sys\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.247275 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.247246    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/946481d5-3192-42a0-949d-6494bc388aa6-metrics-client-ca\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.247403 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.247379    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.248973 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.248954    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-tls\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.249074 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.248983    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/946481d5-3192-42a0-949d-6494bc388aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.283488 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.283452    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9sk4\" (UniqueName: \"kubernetes.io/projected/946481d5-3192-42a0-949d-6494bc388aa6-kube-api-access-r9sk4\") pod \"node-exporter-j59w7\" (UID: \"946481d5-3192-42a0-949d-6494bc388aa6\") " pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.304857 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.304820    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:27.306703 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.306686    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j59w7"
Jul 13 18:52:27.318910 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:27.318886    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946481d5_3192_42a0_949d_6494bc388aa6.slice/crio-c9dcc419509ce4ba1cec77630333c0dbe4b5bb47e0895b6c94ec8dc79900d733 WatchSource:0}: Error finding container c9dcc419509ce4ba1cec77630333c0dbe4b5bb47e0895b6c94ec8dc79900d733: Status 404 returned error can't find the container with id c9dcc419509ce4ba1cec77630333c0dbe4b5bb47e0895b6c94ec8dc79900d733
Jul 13 18:52:27.400904 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.400809    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:27.902053 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.902020    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j59w7" event={"ID":"946481d5-3192-42a0-949d-6494bc388aa6","Type":"ContainerStarted","Data":"c9dcc419509ce4ba1cec77630333c0dbe4b5bb47e0895b6c94ec8dc79900d733"}
Jul 13 18:52:27.915222 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:27.915172    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:28.015118 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:28.015083    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:28.130139 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:28.130104    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:28.229096 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:28.229018    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:28.907159 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:28.907075    2587 generic.go:356] "Generic (PLEG): container finished" podID="946481d5-3192-42a0-949d-6494bc388aa6" containerID="cd7ffb5f93c8c1915c0ff49417d43b5d927b2e850b796b8c7d442e57b178ccdc" exitCode=0
Jul 13 18:52:28.907159 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:28.907143    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j59w7" event={"ID":"946481d5-3192-42a0-949d-6494bc388aa6","Type":"ContainerDied","Data":"cd7ffb5f93c8c1915c0ff49417d43b5d927b2e850b796b8c7d442e57b178ccdc"}
Jul 13 18:52:29.866320 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:29.866271    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:29.912610 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:29.912581    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j59w7" event={"ID":"946481d5-3192-42a0-949d-6494bc388aa6","Type":"ContainerStarted","Data":"a03ad51cb0cc97b6a57389cac0db38284050398c5019ec20d0691ebbaf6444c5"}
Jul 13 18:52:29.912937 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:29.912630    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j59w7" event={"ID":"946481d5-3192-42a0-949d-6494bc388aa6","Type":"ContainerStarted","Data":"9235d1925fac82027cb077d4d0fc2cd47b85e08363e5614fcb319222b8822561"}
Jul 13 18:52:29.939630 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:29.939583    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j59w7" podStartSLOduration=2.784732722 podStartE2EDuration="3.939567638s" totalImagesPullingTime="1.092867331s" totalInitContainerRuntime="61.967585ms" isStatefulPod=true podCreationTimestamp="2026-07-13 18:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:29.937610564 +0000 UTC m=+96.177910655" watchObservedRunningTime="2026-07-13 18:52:29.939567638 +0000 UTC m=+96.179867737"
Jul 13 18:52:29.941360 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:29.941331    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:31.028977 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:31.028941    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:31.091213 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:31.091178    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:31.473958 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:31.473854    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:31.730890 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:31.730811    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:31.821229 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:31.821182    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:31.892382 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:31.892347    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:31.997021 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:31.996935    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:32.786935 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:32.786903    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xvd96"
Jul 13 18:52:33.162059 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:33.161977    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:35.195218 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:35.195158    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:35.292571 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:35.292530    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:38.940984 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:38.940947    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:39.039505 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:39.039458    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:39.223099 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:39.223017    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:39.295009 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:39.294976    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:41.480688 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:41.480648    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:41.581471 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:41.581434    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:42.957742 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:42.957710    2587 generic.go:356] "Generic (PLEG): container finished" podID="6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68" containerID="f41c8c196255d27f0bc55c1dbc24cb81829538f2848d6f40f5fc335e933c5dc6" exitCode=0
Jul 13 18:52:42.958108 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:42.957784    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq" event={"ID":"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68","Type":"ContainerDied","Data":"f41c8c196255d27f0bc55c1dbc24cb81829538f2848d6f40f5fc335e933c5dc6"}
Jul 13 18:52:42.958108 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:42.958073    2587 scope.go:117] "RemoveContainer" containerID="f41c8c196255d27f0bc55c1dbc24cb81829538f2848d6f40f5fc335e933c5dc6"
Jul 13 18:52:43.963424 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:43.963390    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vcftq" event={"ID":"6bbc624c-d8ec-49bb-99ab-a9f9ff7b8b68","Type":"ContainerStarted","Data":"ad3bccac3e1987b548091dd5ecc7ae87c05a3fc42d5b8dd13ffaf4e5193d8e2f"}
Jul 13 18:52:45.881333 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:45.881301    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:45.982540 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:45.982507    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:47.052361 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:47.052320    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:47.108326 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:47.108286    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:48.671776 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.671737    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ccdb84484-kw4gj"]
Jul 13 18:52:48.678800 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.678774    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.681786 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.681767    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 13 18:52:48.681909 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.681768    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 13 18:52:48.682918 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.682904    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 13 18:52:48.682918 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.682909    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xzzm2\""
Jul 13 18:52:48.683031 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.682909    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 13 18:52:48.683031 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.682979    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 13 18:52:48.683295 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.683282    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 13 18:52:48.683352 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.683315    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 13 18:52:48.687995 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.687957    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 13 18:52:48.688702 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.688682    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccdb84484-kw4gj"]
Jul 13 18:52:48.702959 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.702934    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-config\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.703091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.702975    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-oauth-config\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.703091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.703007    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-oauth-serving-cert\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.703091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.703039    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-serving-cert\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.703091 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.703068    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-trusted-ca-bundle\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.703336 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.703228    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gd6\" (UniqueName: \"kubernetes.io/projected/7c94061f-7381-4112-81b3-fd986ff1f6bd-kube-api-access-c4gd6\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.703336 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.703266    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-service-ca\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.804213 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.804175    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-config\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.804372 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.804244    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-oauth-config\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.804372 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.804275    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-oauth-serving-cert\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.804372 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.804303    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-serving-cert\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.804520 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.804438    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-trusted-ca-bundle\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.804571 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.804524    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gd6\" (UniqueName: \"kubernetes.io/projected/7c94061f-7381-4112-81b3-fd986ff1f6bd-kube-api-access-c4gd6\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.804571 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.804558    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-service-ca\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.805060 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.805032    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-config\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.805164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.805082    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-oauth-serving-cert\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.805276 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.805258    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-trusted-ca-bundle\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.805343 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.805321    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-service-ca\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.806830 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.806803    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-oauth-config\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.806830 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.806819    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-serving-cert\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.814142 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.814112    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gd6\" (UniqueName: \"kubernetes.io/projected/7c94061f-7381-4112-81b3-fd986ff1f6bd-kube-api-access-c4gd6\") pod \"console-ccdb84484-kw4gj\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") " pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:48.988856 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:48.988767    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:49.086785 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:49.086754    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:49.115305 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:49.115278    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccdb84484-kw4gj"]
Jul 13 18:52:49.117703 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:52:49.117677    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c94061f_7381_4112_81b3_fd986ff1f6bd.slice/crio-9c784074999b4d0ec187c66b31c96ed0dcfa27211df23596cfa44e26f20eb522 WatchSource:0}: Error finding container 9c784074999b4d0ec187c66b31c96ed0dcfa27211df23596cfa44e26f20eb522: Status 404 returned error can't find the container with id 9c784074999b4d0ec187c66b31c96ed0dcfa27211df23596cfa44e26f20eb522
Jul 13 18:52:49.187520 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:49.187484    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:49.985751 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:49.985712    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccdb84484-kw4gj" event={"ID":"7c94061f-7381-4112-81b3-fd986ff1f6bd","Type":"ContainerStarted","Data":"9c784074999b4d0ec187c66b31c96ed0dcfa27211df23596cfa44e26f20eb522"}
Jul 13 18:52:51.687505 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:51.687462    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:51.792064 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:51.792030    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:52.999215 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:52.999166    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccdb84484-kw4gj" event={"ID":"7c94061f-7381-4112-81b3-fd986ff1f6bd","Type":"ContainerStarted","Data":"e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9"}
Jul 13 18:52:53.024233 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:53.024164    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ccdb84484-kw4gj" podStartSLOduration=1.331126791 podStartE2EDuration="5.024150171s" totalImagesPullingTime="3.69302338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:53.021923495 +0000 UTC m=+119.262223596" watchObservedRunningTime="2026-07-13 18:52:53.024150171 +0000 UTC m=+119.264450270"
Jul 13 18:52:53.343053 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:53.343016    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:53.430541 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:53.430509    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:57.295645 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:57.295611    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:58.798360 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:58.798316    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:58.989834 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:58.989796    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:58.990006 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:58.989859    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:58.994646 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:58.994623    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:59.021619 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:59.021596    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:52:59.661030 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:59.660994    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:59.727948 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:52:59.727911    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:00.842323 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:00.842298    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j59w7_946481d5-3192-42a0-949d-6494bc388aa6/init-textfile/0.log"
Jul 13 18:53:01.043897 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:01.043863    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j59w7_946481d5-3192-42a0-949d-6494bc388aa6/node-exporter/0.log"
Jul 13 18:53:01.243252 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:01.243150    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j59w7_946481d5-3192-42a0-949d-6494bc388aa6/kube-rbac-proxy/0.log"
Jul 13 18:53:03.145680 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:03.145650    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:03.201650 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:03.201605    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:06.449355 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:06.449325    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-hrpdn_19ecb11a-0f10-49d6-ad5e-ce2f00397c0f/networking-console-plugin/0.log"
Jul 13 18:53:07.245313 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:07.245285    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-ccdb84484-kw4gj_7c94061f-7381-4112-81b3-fd986ff1f6bd/console/0.log"
Jul 13 18:53:07.643275 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:07.643243    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c6c7c4594-qv9w2_8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7/router/0.log"
Jul 13 18:53:08.242080 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:08.242050    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kcvql_66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2/serve-healthcheck-canary/0.log"
Jul 13 18:53:10.294550 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:10.294514    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:10.404718 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:10.404688    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:10.503866 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:10.503825    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:17.565817 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:17.565783    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:17.666450 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:17.666415    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:17.723003 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:17.722959    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:26.212919 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:26.212875    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:43.025501 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:43.025439    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:43.090118 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:43.090073    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:48.237451 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:48.237352    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:48.294626 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:48.294589    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:48.431714 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:48.431676    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:48.493833 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:48.493745    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:48.986824 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:48.986783    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:49.039450 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:49.039412    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:54.927095 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:54.927055    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:55.050242 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:55.050188    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:59.270433 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:59.270397    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ccdb84484-kw4gj"]
Jul 13 18:53:59.394402 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:59.394356    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:59.496724 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:59.496679    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:59.597029 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:53:59.596938    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:03.174513 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:03.174475    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:03.275316 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:03.275277    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:24.291549 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.291474    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ccdb84484-kw4gj" podUID="7c94061f-7381-4112-81b3-fd986ff1f6bd" containerName="console" containerID="cri-o://e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9" gracePeriod=15
Jul 13 18:54:24.543739 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.543680    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:54:24.608350 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608315    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-trusted-ca-bundle\") pod \"7c94061f-7381-4112-81b3-fd986ff1f6bd\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") "
Jul 13 18:54:24.608530 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608364    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-service-ca\") pod \"7c94061f-7381-4112-81b3-fd986ff1f6bd\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") "
Jul 13 18:54:24.608530 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608392    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-config\") pod \"7c94061f-7381-4112-81b3-fd986ff1f6bd\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") "
Jul 13 18:54:24.608530 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608432    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4gd6\" (UniqueName: \"kubernetes.io/projected/7c94061f-7381-4112-81b3-fd986ff1f6bd-kube-api-access-c4gd6\") pod \"7c94061f-7381-4112-81b3-fd986ff1f6bd\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") "
Jul 13 18:54:24.608530 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608464    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-oauth-config\") pod \"7c94061f-7381-4112-81b3-fd986ff1f6bd\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") "
Jul 13 18:54:24.608530 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608485    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-serving-cert\") pod \"7c94061f-7381-4112-81b3-fd986ff1f6bd\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") "
Jul 13 18:54:24.608530 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608514    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-oauth-serving-cert\") pod \"7c94061f-7381-4112-81b3-fd986ff1f6bd\" (UID: \"7c94061f-7381-4112-81b3-fd986ff1f6bd\") "
Jul 13 18:54:24.608823 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608768    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7c94061f-7381-4112-81b3-fd986ff1f6bd" (UID: "7c94061f-7381-4112-81b3-fd986ff1f6bd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:54:24.608899 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608869    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c94061f-7381-4112-81b3-fd986ff1f6bd" (UID: "7c94061f-7381-4112-81b3-fd986ff1f6bd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:54:24.608969 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.608952    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-config" (OuterVolumeSpecName: "console-config") pod "7c94061f-7381-4112-81b3-fd986ff1f6bd" (UID: "7c94061f-7381-4112-81b3-fd986ff1f6bd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:54:24.609082 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.609052    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c94061f-7381-4112-81b3-fd986ff1f6bd" (UID: "7c94061f-7381-4112-81b3-fd986ff1f6bd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:54:24.610823 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.610791    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c94061f-7381-4112-81b3-fd986ff1f6bd-kube-api-access-c4gd6" (OuterVolumeSpecName: "kube-api-access-c4gd6") pod "7c94061f-7381-4112-81b3-fd986ff1f6bd" (UID: "7c94061f-7381-4112-81b3-fd986ff1f6bd"). InnerVolumeSpecName "kube-api-access-c4gd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:54:24.610938 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.610876    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c94061f-7381-4112-81b3-fd986ff1f6bd" (UID: "7c94061f-7381-4112-81b3-fd986ff1f6bd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:54:24.610938 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.610902    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c94061f-7381-4112-81b3-fd986ff1f6bd" (UID: "7c94061f-7381-4112-81b3-fd986ff1f6bd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:54:24.709925 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.709883    2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:54:24.709925 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.709917    2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:54:24.709925 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.709930    2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:54:24.710164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.709941    2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:54:24.710164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.709955    2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:54:24.710164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.709967    2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c94061f-7381-4112-81b3-fd986ff1f6bd-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:54:24.710164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:24.709977    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gd6\" (UniqueName: \"kubernetes.io/projected/7c94061f-7381-4112-81b3-fd986ff1f6bd-kube-api-access-c4gd6\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:54:25.314723 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.314686    2587 generic.go:356] "Generic (PLEG): container finished" podID="7c94061f-7381-4112-81b3-fd986ff1f6bd" containerID="e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9" exitCode=0
Jul 13 18:54:25.315114 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.314731    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccdb84484-kw4gj"
Jul 13 18:54:25.315114 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.314732    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccdb84484-kw4gj" event={"ID":"7c94061f-7381-4112-81b3-fd986ff1f6bd","Type":"ContainerDied","Data":"e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9"}
Jul 13 18:54:25.315114 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.314865    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccdb84484-kw4gj" event={"ID":"7c94061f-7381-4112-81b3-fd986ff1f6bd","Type":"ContainerDied","Data":"9c784074999b4d0ec187c66b31c96ed0dcfa27211df23596cfa44e26f20eb522"}
Jul 13 18:54:25.315114 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.314896    2587 scope.go:117] "RemoveContainer" containerID="e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9"
Jul 13 18:54:25.323711 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.323694    2587 scope.go:117] "RemoveContainer" containerID="e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9"
Jul 13 18:54:25.323955 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:54:25.323936    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9\": container with ID starting with e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9 not found: ID does not exist" containerID="e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9"
Jul 13 18:54:25.323997 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.323964    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9"} err="failed to get container status \"e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9\": rpc error: code = NotFound desc = could not find container \"e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9\": container with ID starting with e98985583b6b6c2137e7c5b893396b7da738a355136ef84fef18fb98a1a8fbe9 not found: ID does not exist"
Jul 13 18:54:25.335968 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.335939    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ccdb84484-kw4gj"]
Jul 13 18:54:25.339640 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:25.339617    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ccdb84484-kw4gj"]
Jul 13 18:54:26.504560 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:26.504529    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c94061f-7381-4112-81b3-fd986ff1f6bd" path="/var/lib/kubelet/pods/7c94061f-7381-4112-81b3-fd986ff1f6bd/volumes"
Jul 13 18:54:26.545718 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:26.545681    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:26.596521 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:26.596484    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:57.406445 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:57.406408    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:58.736458 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.736417    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"]
Jul 13 18:54:58.736835 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.736681    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c94061f-7381-4112-81b3-fd986ff1f6bd" containerName="console"
Jul 13 18:54:58.736835 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.736691    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c94061f-7381-4112-81b3-fd986ff1f6bd" containerName="console"
Jul 13 18:54:58.736835 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.736735    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c94061f-7381-4112-81b3-fd986ff1f6bd" containerName="console"
Jul 13 18:54:58.741030 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.741010    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.743944 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.743917    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-whjp9\""
Jul 13 18:54:58.744072 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.743917    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 13 18:54:58.744072 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.743930    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 13 18:54:58.748940 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.748916    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"]
Jul 13 18:54:58.841569 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.841524    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.841569 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.841575    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.841774 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.841603    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svcnc\" (UniqueName: \"kubernetes.io/projected/5f2f6c00-4a14-4a8e-947a-b120be9235e3-kube-api-access-svcnc\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.942802 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.942756    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.942802 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.942806    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.943011 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.942915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcnc\" (UniqueName: \"kubernetes.io/projected/5f2f6c00-4a14-4a8e-947a-b120be9235e3-kube-api-access-svcnc\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.943138 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.943122    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.943226 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.943158    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:58.951473 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:58.951451    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svcnc\" (UniqueName: \"kubernetes.io/projected/5f2f6c00-4a14-4a8e-947a-b120be9235e3-kube-api-access-svcnc\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:59.050844 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:59.050807    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:54:59.179980 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:59.179948    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"]
Jul 13 18:54:59.183385 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:54:59.183356    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2f6c00_4a14_4a8e_947a_b120be9235e3.slice/crio-9a2ea0c9b6999de4c84decb3c65742a3d7002d2991392b881a8ab7f5eeea4d07 WatchSource:0}: Error finding container 9a2ea0c9b6999de4c84decb3c65742a3d7002d2991392b881a8ab7f5eeea4d07: Status 404 returned error can't find the container with id 9a2ea0c9b6999de4c84decb3c65742a3d7002d2991392b881a8ab7f5eeea4d07
Jul 13 18:54:59.433093 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:54:59.433004    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr" event={"ID":"5f2f6c00-4a14-4a8e-947a-b120be9235e3","Type":"ContainerStarted","Data":"9a2ea0c9b6999de4c84decb3c65742a3d7002d2991392b881a8ab7f5eeea4d07"}
Jul 13 18:55:00.393809 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:00.393768    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:00.454093 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:00.454053    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:05.458396 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:05.458360    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:09.469677 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:09.469644    2587 generic.go:356] "Generic (PLEG): container finished" podID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerID="e366993395234be6b24479cc5d61f84fd3e3d29209d6143bc59d40984bba3947" exitCode=0
Jul 13 18:55:09.470043 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:09.469716    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr" event={"ID":"5f2f6c00-4a14-4a8e-947a-b120be9235e3","Type":"ContainerDied","Data":"e366993395234be6b24479cc5d61f84fd3e3d29209d6143bc59d40984bba3947"}
Jul 13 18:55:11.478798 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:11.478755    2587 generic.go:356] "Generic (PLEG): container finished" podID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerID="4c1060098e4358e5a8bc08387692269015552aade241317dfed2ece5ca5326c3" exitCode=0
Jul 13 18:55:11.479293 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:11.478841    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr" event={"ID":"5f2f6c00-4a14-4a8e-947a-b120be9235e3","Type":"ContainerDied","Data":"4c1060098e4358e5a8bc08387692269015552aade241317dfed2ece5ca5326c3"}
Jul 13 18:55:21.517429 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:21.517394    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr" event={"ID":"5f2f6c00-4a14-4a8e-947a-b120be9235e3","Type":"ContainerStarted","Data":"644763b8196019d8de73f37542573787946182c860c4efce5c5fcee478765ebd"}
Jul 13 18:55:21.537404 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:21.537354    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr" podStartSLOduration=2.580996679 podStartE2EDuration="23.537337159s" totalImagesPullingTime="20.95634048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:54:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:55:21.536709337 +0000 UTC m=+267.777009438" watchObservedRunningTime="2026-07-13 18:55:21.537337159 +0000 UTC m=+267.777637259"
Jul 13 18:55:22.522511 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:22.522476    2587 generic.go:356] "Generic (PLEG): container finished" podID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerID="644763b8196019d8de73f37542573787946182c860c4efce5c5fcee478765ebd" exitCode=0
Jul 13 18:55:22.522962 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:22.522527    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr" event={"ID":"5f2f6c00-4a14-4a8e-947a-b120be9235e3","Type":"ContainerDied","Data":"644763b8196019d8de73f37542573787946182c860c4efce5c5fcee478765ebd"}
Jul 13 18:55:23.659191 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.659159    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:55:23.738545 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.738506    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-bundle\") pod \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") "
Jul 13 18:55:23.738719 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.738590    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-util\") pod \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") "
Jul 13 18:55:23.738719 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.738622    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svcnc\" (UniqueName: \"kubernetes.io/projected/5f2f6c00-4a14-4a8e-947a-b120be9235e3-kube-api-access-svcnc\") pod \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\" (UID: \"5f2f6c00-4a14-4a8e-947a-b120be9235e3\") "
Jul 13 18:55:23.739300 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.739260    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-bundle" (OuterVolumeSpecName: "bundle") pod "5f2f6c00-4a14-4a8e-947a-b120be9235e3" (UID: "5f2f6c00-4a14-4a8e-947a-b120be9235e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:55:23.740907 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.740877    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2f6c00-4a14-4a8e-947a-b120be9235e3-kube-api-access-svcnc" (OuterVolumeSpecName: "kube-api-access-svcnc") pod "5f2f6c00-4a14-4a8e-947a-b120be9235e3" (UID: "5f2f6c00-4a14-4a8e-947a-b120be9235e3"). InnerVolumeSpecName "kube-api-access-svcnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:55:23.744282 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.744260    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-util" (OuterVolumeSpecName: "util") pod "5f2f6c00-4a14-4a8e-947a-b120be9235e3" (UID: "5f2f6c00-4a14-4a8e-947a-b120be9235e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:55:23.839463 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.839381    2587 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:55:23.839463 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.839413    2587 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2f6c00-4a14-4a8e-947a-b120be9235e3-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:55:23.839463 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:23.839423    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svcnc\" (UniqueName: \"kubernetes.io/projected/5f2f6c00-4a14-4a8e-947a-b120be9235e3-kube-api-access-svcnc\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:55:24.530873 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:24.530840    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr" event={"ID":"5f2f6c00-4a14-4a8e-947a-b120be9235e3","Type":"ContainerDied","Data":"9a2ea0c9b6999de4c84decb3c65742a3d7002d2991392b881a8ab7f5eeea4d07"}
Jul 13 18:55:24.531046 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:24.530897    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a2ea0c9b6999de4c84decb3c65742a3d7002d2991392b881a8ab7f5eeea4d07"
Jul 13 18:55:24.531046 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:24.530853    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lt2dr"
Jul 13 18:55:25.479687 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:25.479653    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:25.576460 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:25.576416    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:54.293164 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:54.293136    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 18:55:54.296362 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:54.296341    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 18:55:54.298804 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:55:54.298781    2587 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 18:56:13.592293 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:13.592252    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:15.371658 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:15.371620    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:20.132989 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.132919    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:20.413402 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.413315    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:20.422532 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.422501    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-mrs5d"]
Jul 13 18:56:20.422747 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.422735    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerName="util"
Jul 13 18:56:20.422787 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.422748    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerName="util"
Jul 13 18:56:20.422787 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.422756    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerName="pull"
Jul 13 18:56:20.422787 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.422762    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerName="pull"
Jul 13 18:56:20.422787 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.422780    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerName="extract"
Jul 13 18:56:20.422787 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.422786    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerName="extract"
Jul 13 18:56:20.422928 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.422828    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2f6c00-4a14-4a8e-947a-b120be9235e3" containerName="extract"
Jul 13 18:56:20.424718 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.424702    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.427302 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.427276    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 13 18:56:20.427432 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.427413    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nc756\""
Jul 13 18:56:20.427513 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.427473    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 13 18:56:20.427513 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.427508    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 13 18:56:20.428458 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.428439    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 13 18:56:20.428550 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.428494    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 13 18:56:20.434820 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.434794    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-mrs5d"]
Jul 13 18:56:20.443260 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.443223    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cceea6ba-717e-499b-b217-63e149ba30f6-cabundle0\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.443372 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.443279    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89686\" (UniqueName: \"kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-kube-api-access-89686\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.443437 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.443407    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-certificates\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.543892 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.543860    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-certificates\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.544102 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.543911    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cceea6ba-717e-499b-b217-63e149ba30f6-cabundle0\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.544102 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:56:20.544000    2587 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 13 18:56:20.544102 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:56:20.544020    2587 secret.go:281] references non-existent secret key: ca.crt
Jul 13 18:56:20.544102 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:56:20.544028    2587 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 13 18:56:20.544102 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:56:20.544040    2587 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-mrs5d: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 13 18:56:20.544102 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.544077    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89686\" (UniqueName: \"kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-kube-api-access-89686\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.544102 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:56:20.544099    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-certificates podName:cceea6ba-717e-499b-b217-63e149ba30f6 nodeName:}" failed. No retries permitted until 2026-07-13 18:56:21.044083911 +0000 UTC m=+327.284383990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-certificates") pod "keda-operator-77f648d694-mrs5d" (UID: "cceea6ba-717e-499b-b217-63e149ba30f6") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 13 18:56:20.544648 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.544627    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cceea6ba-717e-499b-b217-63e149ba30f6-cabundle0\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.552912 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.552891    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89686\" (UniqueName: \"kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-kube-api-access-89686\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:20.603144 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.603109    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:20.793257 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.793197    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:20.957735 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:20.957700    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:21.048515 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.048424    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-certificates\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:21.050824 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.050792    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceea6ba-717e-499b-b217-63e149ba30f6-certificates\") pod \"keda-operator-77f648d694-mrs5d\" (UID: \"cceea6ba-717e-499b-b217-63e149ba30f6\") " pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:21.055351 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.055277    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:21.175793 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.175761    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:21.287437 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.287396    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:21.335669 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.335584    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:21.377147 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.377116    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:21.463751 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.463695    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-mrs5d"]
Jul 13 18:56:21.465867 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:56:21.465844    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcceea6ba_717e_499b_b217_63e149ba30f6.slice/crio-8bad8ce48717177c0c96e138f5b885c4c2bcc70b1f04c58b5749ae36c34756e0 WatchSource:0}: Error finding container 8bad8ce48717177c0c96e138f5b885c4c2bcc70b1f04c58b5749ae36c34756e0: Status 404 returned error can't find the container with id 8bad8ce48717177c0c96e138f5b885c4c2bcc70b1f04c58b5749ae36c34756e0
Jul 13 18:56:21.467077 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.467059    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:56:21.533688 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.533640    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:21.632685 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.632590    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:21.722530 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:21.722487    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-mrs5d" event={"ID":"cceea6ba-717e-499b-b217-63e149ba30f6","Type":"ContainerStarted","Data":"8bad8ce48717177c0c96e138f5b885c4c2bcc70b1f04c58b5749ae36c34756e0"}
Jul 13 18:56:26.215524 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:26.215485    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:26.742639 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:26.742598    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-mrs5d" event={"ID":"cceea6ba-717e-499b-b217-63e149ba30f6","Type":"ContainerStarted","Data":"70a8199365e017d6ada2717f5efd3af083c3179355f754dbef2c5979b2617dc1"}
Jul 13 18:56:26.742815 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:26.742743    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:56:26.764778 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:26.764723    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-mrs5d" podStartSLOduration=2.534443058 podStartE2EDuration="6.764704243s" totalImagesPullingTime="4.230261185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:56:26.762563025 +0000 UTC m=+333.002863126" watchObservedRunningTime="2026-07-13 18:56:26.764704243 +0000 UTC m=+333.005004343"
Jul 13 18:56:47.748893 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:56:47.748818    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-mrs5d"
Jul 13 18:57:25.722970 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.722932    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:25.787699 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.787662    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-4rsww"]
Jul 13 18:57:25.789809 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.789788    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:25.793581 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.793555    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-gn7dw\""
Jul 13 18:57:25.793805 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.793787    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 13 18:57:25.794674 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.794655    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 13 18:57:25.794766 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.794712    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 13 18:57:25.802515 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.802478    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-4rsww"]
Jul 13 18:57:25.830152 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.830116    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-8pwhq"]
Jul 13 18:57:25.832338 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.832320    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:25.835023 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.835000    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-knvzt\""
Jul 13 18:57:25.835362 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.835347    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 13 18:57:25.845990 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.845964    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8pwhq"]
Jul 13 18:57:25.916318 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.916287    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert\") pod \"kserve-controller-manager-6b69b6d547-4rsww\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") " pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:25.916497 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:25.916337    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttcv5\" (UniqueName: \"kubernetes.io/projected/2d825f62-3806-48be-adcf-f8f8befb20af-kube-api-access-ttcv5\") pod \"kserve-controller-manager-6b69b6d547-4rsww\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") " pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:26.017088 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.016996    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttcv5\" (UniqueName: \"kubernetes.io/projected/2d825f62-3806-48be-adcf-f8f8befb20af-kube-api-access-ttcv5\") pod \"kserve-controller-manager-6b69b6d547-4rsww\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") " pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:26.017088 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.017045    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc8a59a0-69d3-4dcb-b09c-fe41735f9319-data\") pod \"seaweedfs-64568bcd49-8pwhq\" (UID: \"bc8a59a0-69d3-4dcb-b09c-fe41735f9319\") " pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:26.017088 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.017085    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert\") pod \"kserve-controller-manager-6b69b6d547-4rsww\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") " pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:26.017330 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.017109    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjnj\" (UniqueName: \"kubernetes.io/projected/bc8a59a0-69d3-4dcb-b09c-fe41735f9319-kube-api-access-bwjnj\") pod \"seaweedfs-64568bcd49-8pwhq\" (UID: \"bc8a59a0-69d3-4dcb-b09c-fe41735f9319\") " pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:26.017330 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:57:26.017229    2587 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 13 18:57:26.017330 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:57:26.017288    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert podName:2d825f62-3806-48be-adcf-f8f8befb20af nodeName:}" failed. No retries permitted until 2026-07-13 18:57:26.517272083 +0000 UTC m=+392.757572162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert") pod "kserve-controller-manager-6b69b6d547-4rsww" (UID: "2d825f62-3806-48be-adcf-f8f8befb20af") : secret "kserve-webhook-server-cert" not found
Jul 13 18:57:26.026100 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.026076    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttcv5\" (UniqueName: \"kubernetes.io/projected/2d825f62-3806-48be-adcf-f8f8befb20af-kube-api-access-ttcv5\") pod \"kserve-controller-manager-6b69b6d547-4rsww\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") " pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:26.118033 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.117995    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjnj\" (UniqueName: \"kubernetes.io/projected/bc8a59a0-69d3-4dcb-b09c-fe41735f9319-kube-api-access-bwjnj\") pod \"seaweedfs-64568bcd49-8pwhq\" (UID: \"bc8a59a0-69d3-4dcb-b09c-fe41735f9319\") " pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:26.118253 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.118051    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc8a59a0-69d3-4dcb-b09c-fe41735f9319-data\") pod \"seaweedfs-64568bcd49-8pwhq\" (UID: \"bc8a59a0-69d3-4dcb-b09c-fe41735f9319\") " pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:26.118435 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.118416    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc8a59a0-69d3-4dcb-b09c-fe41735f9319-data\") pod \"seaweedfs-64568bcd49-8pwhq\" (UID: \"bc8a59a0-69d3-4dcb-b09c-fe41735f9319\") " pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:26.126141 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.126109    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjnj\" (UniqueName: \"kubernetes.io/projected/bc8a59a0-69d3-4dcb-b09c-fe41735f9319-kube-api-access-bwjnj\") pod \"seaweedfs-64568bcd49-8pwhq\" (UID: \"bc8a59a0-69d3-4dcb-b09c-fe41735f9319\") " pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:26.141489 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.141463    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:26.156988 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.156958    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:26.248605 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.248573    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:26.267556 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.267482    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8pwhq"]
Jul 13 18:57:26.270387 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:57:26.270354    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8a59a0_69d3_4dcb_b09c_fe41735f9319.slice/crio-f76f0358996953ae0e9af66ac7588258574a3c630092c1c4ed5c46e66e5f7b60 WatchSource:0}: Error finding container f76f0358996953ae0e9af66ac7588258574a3c630092c1c4ed5c46e66e5f7b60: Status 404 returned error can't find the container with id f76f0358996953ae0e9af66ac7588258574a3c630092c1c4ed5c46e66e5f7b60
Jul 13 18:57:26.521720 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.521693    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert\") pod \"kserve-controller-manager-6b69b6d547-4rsww\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") " pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:26.524357 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.524326    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert\") pod \"kserve-controller-manager-6b69b6d547-4rsww\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") " pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:26.700525 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.700485    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:26.723284 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.723247    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:26.824012 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.823969    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:26.853933 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.853888    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-4rsww"]
Jul 13 18:57:26.882359 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.882306    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:26.918283 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:57:26.918245    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d825f62_3806_48be_adcf_f8f8befb20af.slice/crio-de60d8e5f4bb46f1eaa204a8673c4aeb04333e9d6e03ca4ed759dce548d607a1 WatchSource:0}: Error finding container de60d8e5f4bb46f1eaa204a8673c4aeb04333e9d6e03ca4ed759dce548d607a1: Status 404 returned error can't find the container with id de60d8e5f4bb46f1eaa204a8673c4aeb04333e9d6e03ca4ed759dce548d607a1
Jul 13 18:57:26.950679 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.950643    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww" event={"ID":"2d825f62-3806-48be-adcf-f8f8befb20af","Type":"ContainerStarted","Data":"de60d8e5f4bb46f1eaa204a8673c4aeb04333e9d6e03ca4ed759dce548d607a1"}
Jul 13 18:57:26.951833 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:26.951808    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8pwhq" event={"ID":"bc8a59a0-69d3-4dcb-b09c-fe41735f9319","Type":"ContainerStarted","Data":"f76f0358996953ae0e9af66ac7588258574a3c630092c1c4ed5c46e66e5f7b60"}
Jul 13 18:57:30.973594 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:30.973504    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww" event={"ID":"2d825f62-3806-48be-adcf-f8f8befb20af","Type":"ContainerStarted","Data":"b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612"}
Jul 13 18:57:30.974013 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:30.973634    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:57:30.975130 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:30.975103    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8pwhq" event={"ID":"bc8a59a0-69d3-4dcb-b09c-fe41735f9319","Type":"ContainerStarted","Data":"c45e5c40215876ab70046ba0209808d1ce7984f5b13c4aa321cea65ed057e289"}
Jul 13 18:57:30.975240 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:30.975226    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:30.992801 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:30.992752    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww" podStartSLOduration=2.384607979 podStartE2EDuration="5.992738629s" totalImagesPullingTime="3.60813065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:57:30.992576714 +0000 UTC m=+397.232876814" watchObservedRunningTime="2026-07-13 18:57:30.992738629 +0000 UTC m=+397.233038772"
Jul 13 18:57:31.010123 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:31.010065    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-8pwhq" podStartSLOduration=1.660042858 podStartE2EDuration="6.010048342s" totalImagesPullingTime="4.350005484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:57:31.009865339 +0000 UTC m=+397.250165442" watchObservedRunningTime="2026-07-13 18:57:31.010048342 +0000 UTC m=+397.250348442"
Jul 13 18:57:36.980709 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:36.980678    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-8pwhq"
Jul 13 18:57:42.706433 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:42.706390    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:42.809959 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:42.809926    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:42.908007 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:42.907967    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:43.471807 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:43.471769    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:43.566827 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:43.566792    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:44.053042 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:44.053003    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:51.418532 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:51.418493    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:52.635889 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:52.635845    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:53.283476 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:53.283410    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:53.344259 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:53.344220    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:53.398902 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:57:53.398870    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:01.984266 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:01.984235    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:58:02.544913 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.544882    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-4rsww"]
Jul 13 18:58:02.545135 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.545099    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww" podUID="2d825f62-3806-48be-adcf-f8f8befb20af" containerName="manager" containerID="cri-o://b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612" gracePeriod=10
Jul 13 18:58:02.567976 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.567947    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-6g2dl"]
Jul 13 18:58:02.570065 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.570050    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:02.581048 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.581016    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-6g2dl"]
Jul 13 18:58:02.683780 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.683744    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b31d809-7dca-44d1-8060-2b7a2be8c41e-cert\") pod \"kserve-controller-manager-6b69b6d547-6g2dl\" (UID: \"4b31d809-7dca-44d1-8060-2b7a2be8c41e\") " pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:02.683938 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.683790    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnkvg\" (UniqueName: \"kubernetes.io/projected/4b31d809-7dca-44d1-8060-2b7a2be8c41e-kube-api-access-nnkvg\") pod \"kserve-controller-manager-6b69b6d547-6g2dl\" (UID: \"4b31d809-7dca-44d1-8060-2b7a2be8c41e\") " pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:02.778376 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.778350    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:58:02.784624 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.784603    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b31d809-7dca-44d1-8060-2b7a2be8c41e-cert\") pod \"kserve-controller-manager-6b69b6d547-6g2dl\" (UID: \"4b31d809-7dca-44d1-8060-2b7a2be8c41e\") " pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:02.784714 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.784642    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnkvg\" (UniqueName: \"kubernetes.io/projected/4b31d809-7dca-44d1-8060-2b7a2be8c41e-kube-api-access-nnkvg\") pod \"kserve-controller-manager-6b69b6d547-6g2dl\" (UID: \"4b31d809-7dca-44d1-8060-2b7a2be8c41e\") " pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:02.786977 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.786953    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b31d809-7dca-44d1-8060-2b7a2be8c41e-cert\") pod \"kserve-controller-manager-6b69b6d547-6g2dl\" (UID: \"4b31d809-7dca-44d1-8060-2b7a2be8c41e\") " pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:02.792735 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.792710    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnkvg\" (UniqueName: \"kubernetes.io/projected/4b31d809-7dca-44d1-8060-2b7a2be8c41e-kube-api-access-nnkvg\") pod \"kserve-controller-manager-6b69b6d547-6g2dl\" (UID: \"4b31d809-7dca-44d1-8060-2b7a2be8c41e\") " pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:02.885870 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.885774    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert\") pod \"2d825f62-3806-48be-adcf-f8f8befb20af\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") "
Jul 13 18:58:02.885870 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.885845    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttcv5\" (UniqueName: \"kubernetes.io/projected/2d825f62-3806-48be-adcf-f8f8befb20af-kube-api-access-ttcv5\") pod \"2d825f62-3806-48be-adcf-f8f8befb20af\" (UID: \"2d825f62-3806-48be-adcf-f8f8befb20af\") "
Jul 13 18:58:02.888051 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.888023    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d825f62-3806-48be-adcf-f8f8befb20af-kube-api-access-ttcv5" (OuterVolumeSpecName: "kube-api-access-ttcv5") pod "2d825f62-3806-48be-adcf-f8f8befb20af" (UID: "2d825f62-3806-48be-adcf-f8f8befb20af"). InnerVolumeSpecName "kube-api-access-ttcv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:58:02.888158 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.888019    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert" (OuterVolumeSpecName: "cert") pod "2d825f62-3806-48be-adcf-f8f8befb20af" (UID: "2d825f62-3806-48be-adcf-f8f8befb20af"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:58:02.929241 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.929189    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:02.987065 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.987007    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttcv5\" (UniqueName: \"kubernetes.io/projected/2d825f62-3806-48be-adcf-f8f8befb20af-kube-api-access-ttcv5\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:58:02.987065 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:02.987040    2587 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d825f62-3806-48be-adcf-f8f8befb20af-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 18:58:03.051211 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.051173    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-6g2dl"]
Jul 13 18:58:03.053537 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:58:03.053505    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b31d809_7dca_44d1_8060_2b7a2be8c41e.slice/crio-987358cec738e3b2d1724e3d2f9c364192acc6d61cee60aaa14b2336ae4f1507 WatchSource:0}: Error finding container 987358cec738e3b2d1724e3d2f9c364192acc6d61cee60aaa14b2336ae4f1507: Status 404 returned error can't find the container with id 987358cec738e3b2d1724e3d2f9c364192acc6d61cee60aaa14b2336ae4f1507
Jul 13 18:58:03.088028 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.087995    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl" event={"ID":"4b31d809-7dca-44d1-8060-2b7a2be8c41e","Type":"ContainerStarted","Data":"987358cec738e3b2d1724e3d2f9c364192acc6d61cee60aaa14b2336ae4f1507"}
Jul 13 18:58:03.089175 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.089150    2587 generic.go:356] "Generic (PLEG): container finished" podID="2d825f62-3806-48be-adcf-f8f8befb20af" containerID="b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612" exitCode=0
Jul 13 18:58:03.089293 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.089196    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww"
Jul 13 18:58:03.089293 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.089221    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww" event={"ID":"2d825f62-3806-48be-adcf-f8f8befb20af","Type":"ContainerDied","Data":"b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612"}
Jul 13 18:58:03.089293 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.089257    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-4rsww" event={"ID":"2d825f62-3806-48be-adcf-f8f8befb20af","Type":"ContainerDied","Data":"de60d8e5f4bb46f1eaa204a8673c4aeb04333e9d6e03ca4ed759dce548d607a1"}
Jul 13 18:58:03.089293 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.089278    2587 scope.go:117] "RemoveContainer" containerID="b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612"
Jul 13 18:58:03.097916 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.097896    2587 scope.go:117] "RemoveContainer" containerID="b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612"
Jul 13 18:58:03.098176 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:58:03.098152    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612\": container with ID starting with b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612 not found: ID does not exist" containerID="b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612"
Jul 13 18:58:03.098265 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.098188    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612"} err="failed to get container status \"b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612\": rpc error: code = NotFound desc = could not find container \"b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612\": container with ID starting with b25d59cd57cd7c3fb8c34195ba80e0d417cffea4f5dc966e25e6cf2c0546f612 not found: ID does not exist"
Jul 13 18:58:03.110939 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.110916    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-4rsww"]
Jul 13 18:58:03.114747 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:03.114721    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-4rsww"]
Jul 13 18:58:04.094735 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:04.094697    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl" event={"ID":"4b31d809-7dca-44d1-8060-2b7a2be8c41e","Type":"ContainerStarted","Data":"1f123d910d39a923d988be66ecf97ce81e6dfbcce96c720bea852d035782840a"}
Jul 13 18:58:04.095140 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:04.094821    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:04.112105 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:04.112049    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl" podStartSLOduration=1.61361212 podStartE2EDuration="2.112032326s" totalImagesPullingTime="498.420206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:58:04.110985795 +0000 UTC m=+430.351285896" watchObservedRunningTime="2026-07-13 18:58:04.112032326 +0000 UTC m=+430.352332435"
Jul 13 18:58:04.508543 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:04.508451    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d825f62-3806-48be-adcf-f8f8befb20af" path="/var/lib/kubelet/pods/2d825f62-3806-48be-adcf-f8f8befb20af/volumes"
Jul 13 18:58:35.103940 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:35.103907    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6b69b6d547-6g2dl"
Jul 13 18:58:36.194750 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.194718    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:36.242150 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.240192    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-7n2qz"]
Jul 13 18:58:36.242150 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.240780    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d825f62-3806-48be-adcf-f8f8befb20af" containerName="manager"
Jul 13 18:58:36.242150 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.240802    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d825f62-3806-48be-adcf-f8f8befb20af" containerName="manager"
Jul 13 18:58:36.242150 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.240920    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d825f62-3806-48be-adcf-f8f8befb20af" containerName="manager"
Jul 13 18:58:36.244519 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.244491    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:36.247162 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.247139    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 13 18:58:36.247278 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.247223    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-c9lnd\""
Jul 13 18:58:36.253904 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.253881    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-7n2qz"]
Jul 13 18:58:36.264732 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.264706    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:36.338241 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.338190    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19803efb-662f-4eb6-9005-e2bdbdecd137-tls-certs\") pod \"model-serving-api-fd65d7d6-7n2qz\" (UID: \"19803efb-662f-4eb6-9005-e2bdbdecd137\") " pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:36.338418 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.338256    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxt8\" (UniqueName: \"kubernetes.io/projected/19803efb-662f-4eb6-9005-e2bdbdecd137-kube-api-access-zdxt8\") pod \"model-serving-api-fd65d7d6-7n2qz\" (UID: \"19803efb-662f-4eb6-9005-e2bdbdecd137\") " pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:36.439544 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.439501    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19803efb-662f-4eb6-9005-e2bdbdecd137-tls-certs\") pod \"model-serving-api-fd65d7d6-7n2qz\" (UID: \"19803efb-662f-4eb6-9005-e2bdbdecd137\") " pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:36.439732 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.439564    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxt8\" (UniqueName: \"kubernetes.io/projected/19803efb-662f-4eb6-9005-e2bdbdecd137-kube-api-access-zdxt8\") pod \"model-serving-api-fd65d7d6-7n2qz\" (UID: \"19803efb-662f-4eb6-9005-e2bdbdecd137\") " pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:36.439732 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:58:36.439658    2587 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 13 18:58:36.439845 ip-10-0-139-100 kubenswrapper[2587]: E0713 18:58:36.439738    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19803efb-662f-4eb6-9005-e2bdbdecd137-tls-certs podName:19803efb-662f-4eb6-9005-e2bdbdecd137 nodeName:}" failed. No retries permitted until 2026-07-13 18:58:36.939720341 +0000 UTC m=+463.180020420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19803efb-662f-4eb6-9005-e2bdbdecd137-tls-certs") pod "model-serving-api-fd65d7d6-7n2qz" (UID: "19803efb-662f-4eb6-9005-e2bdbdecd137") : secret "model-serving-api-tls" not found
Jul 13 18:58:36.449768 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.449700    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxt8\" (UniqueName: \"kubernetes.io/projected/19803efb-662f-4eb6-9005-e2bdbdecd137-kube-api-access-zdxt8\") pod \"model-serving-api-fd65d7d6-7n2qz\" (UID: \"19803efb-662f-4eb6-9005-e2bdbdecd137\") " pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:36.555577 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.555544    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:36.808818 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.808781    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:36.944482 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.944442    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19803efb-662f-4eb6-9005-e2bdbdecd137-tls-certs\") pod \"model-serving-api-fd65d7d6-7n2qz\" (UID: \"19803efb-662f-4eb6-9005-e2bdbdecd137\") " pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:36.946957 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:36.946926    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19803efb-662f-4eb6-9005-e2bdbdecd137-tls-certs\") pod \"model-serving-api-fd65d7d6-7n2qz\" (UID: \"19803efb-662f-4eb6-9005-e2bdbdecd137\") " pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:37.155191 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:37.155090    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:37.281045 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:37.280980    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-7n2qz"]
Jul 13 18:58:37.286355 ip-10-0-139-100 kubenswrapper[2587]: W0713 18:58:37.286324    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19803efb_662f_4eb6_9005_e2bdbdecd137.slice/crio-32d7db6e9ea772b0dfab7b50ab227e4aaafafb6f5e8839cf904bca6f81135ab8 WatchSource:0}: Error finding container 32d7db6e9ea772b0dfab7b50ab227e4aaafafb6f5e8839cf904bca6f81135ab8: Status 404 returned error can't find the container with id 32d7db6e9ea772b0dfab7b50ab227e4aaafafb6f5e8839cf904bca6f81135ab8
Jul 13 18:58:37.360646 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:37.360609    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:37.419259 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:37.419155    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:38.216192 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:38.216158    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-7n2qz" event={"ID":"19803efb-662f-4eb6-9005-e2bdbdecd137","Type":"ContainerStarted","Data":"32d7db6e9ea772b0dfab7b50ab227e4aaafafb6f5e8839cf904bca6f81135ab8"}
Jul 13 18:58:40.225266 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:40.225229    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-7n2qz" event={"ID":"19803efb-662f-4eb6-9005-e2bdbdecd137","Type":"ContainerStarted","Data":"f8e24595cfaacfbf57ef8434c559b20f13e06e612939dcad6c05aa98b7422448"}
Jul 13 18:58:40.225665 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:40.225359    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:58:40.244391 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:40.244338    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-7n2qz" podStartSLOduration=2.003491601 podStartE2EDuration="4.244319771s" totalImagesPullingTime="2.24082817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:58:40.242662882 +0000 UTC m=+466.482962982" watchObservedRunningTime="2026-07-13 18:58:40.244319771 +0000 UTC m=+466.484619870"
Jul 13 18:58:51.233606 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:58:51.233575    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-7n2qz"
Jul 13 18:59:05.824059 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:59:05.824017    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:18.114058 ip-10-0-139-100 kubenswrapper[2587]: I0713 18:59:18.114024    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:12.601675 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:12.601589    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:13.593511 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:13.593470    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:24.428960 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.428920    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:24.685572 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.685493    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:24.768179 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.768143    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"]
Jul 13 19:00:24.771443 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.771418    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.777825 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.777659    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-19366-predictor-serving-cert\""
Jul 13 19:00:24.777984 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.777957    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-19366-kube-rbac-proxy-sar-config\""
Jul 13 19:00:24.778071 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.777964    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:00:24.779031 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.779013    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-98mz6\""
Jul 13 19:00:24.779124 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.779017    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:00:24.797220 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.795658    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"]
Jul 13 19:00:24.853254 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.853195    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.853413 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.853285    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-proxy-tls\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.853413 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.853315    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwplp\" (UniqueName: \"kubernetes.io/projected/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-api-access-xwplp\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.954494 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.954406    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-proxy-tls\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.954494 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.954453    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwplp\" (UniqueName: \"kubernetes.io/projected/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-api-access-xwplp\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.954705 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.954517    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.955218 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.955184    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.957096 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.957071    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-proxy-tls\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:24.965631 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:24.965607    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwplp\" (UniqueName: \"kubernetes.io/projected/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-api-access-xwplp\") pod \"success-200-isvc-19366-predictor-6b66d49548-wrwt5\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") " pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:25.039363 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.039325    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"]
Jul 13 19:00:25.042621 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.042600    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.045014 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.044990    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-19366-predictor-serving-cert\""
Jul 13 19:00:25.045100 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.045015    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-19366-kube-rbac-proxy-sar-config\""
Jul 13 19:00:25.052551 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.052528    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"]
Jul 13 19:00:25.055156 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.055132    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.055277 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.055236    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rz76\" (UniqueName: \"kubernetes.io/projected/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-api-access-5rz76\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.055341 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.055313    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.081672 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.081644    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:25.155639 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.155610    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rz76\" (UniqueName: \"kubernetes.io/projected/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-api-access-5rz76\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.155786 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.155665    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.155786 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.155693    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.155903 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:00:25.155810    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-19366-predictor-serving-cert: secret "error-404-isvc-19366-predictor-serving-cert" not found
Jul 13 19:00:25.155903 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:00:25.155890    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls podName:9bf5f463-4192-4760-8fff-33b41a409c0e nodeName:}" failed. No retries permitted until 2026-07-13 19:00:25.655868709 +0000 UTC m=+571.896168796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls") pod "error-404-isvc-19366-predictor-758f8cb57-swfg9" (UID: "9bf5f463-4192-4760-8fff-33b41a409c0e") : secret "error-404-isvc-19366-predictor-serving-cert" not found
Jul 13 19:00:25.156459 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.156435    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.165682 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.165657    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rz76\" (UniqueName: \"kubernetes.io/projected/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-api-access-5rz76\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.212458 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.212346    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:25.224295 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.224272    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"]
Jul 13 19:00:25.225998 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:00:25.225973    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e489e9b_a37d_4a7f_a8f4_8f53153883c3.slice/crio-b61e8dc0422b1873eecbf926eae420370c4445630cb726ac11aa309c10f42997 WatchSource:0}: Error finding container b61e8dc0422b1873eecbf926eae420370c4445630cb726ac11aa309c10f42997: Status 404 returned error can't find the container with id b61e8dc0422b1873eecbf926eae420370c4445630cb726ac11aa309c10f42997
Jul 13 19:00:25.614970 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.614927    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" event={"ID":"6e489e9b-a37d-4a7f-a8f4-8f53153883c3","Type":"ContainerStarted","Data":"b61e8dc0422b1873eecbf926eae420370c4445630cb726ac11aa309c10f42997"}
Jul 13 19:00:25.659724 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.659685    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.661991 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.661967    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls\") pod \"error-404-isvc-19366-predictor-758f8cb57-swfg9\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:25.662866 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.662842    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:25.954238 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:25.954120    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:26.022908 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:26.022873    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:26.103381 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:26.103259    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"]
Jul 13 19:00:26.105789 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:00:26.105762    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf5f463_4192_4760_8fff_33b41a409c0e.slice/crio-c8fe03e561b0ab3ad888f72c19c726e86ed6c36e04ac0afae5a7306226638f19 WatchSource:0}: Error finding container c8fe03e561b0ab3ad888f72c19c726e86ed6c36e04ac0afae5a7306226638f19: Status 404 returned error can't find the container with id c8fe03e561b0ab3ad888f72c19c726e86ed6c36e04ac0afae5a7306226638f19
Jul 13 19:00:26.623270 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:26.623154    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" event={"ID":"9bf5f463-4192-4760-8fff-33b41a409c0e","Type":"ContainerStarted","Data":"c8fe03e561b0ab3ad888f72c19c726e86ed6c36e04ac0afae5a7306226638f19"}
Jul 13 19:00:34.890221 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:34.890162    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:41.714353 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:41.714256    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" event={"ID":"9bf5f463-4192-4760-8fff-33b41a409c0e","Type":"ContainerStarted","Data":"03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2"}
Jul 13 19:00:41.715712 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:41.715681    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" event={"ID":"6e489e9b-a37d-4a7f-a8f4-8f53153883c3","Type":"ContainerStarted","Data":"5891c325999604fb984529e829aa8237a4d09c6b0f1ea4c2601878fee860b606"}
Jul 13 19:00:44.730265 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:44.730221    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" event={"ID":"9bf5f463-4192-4760-8fff-33b41a409c0e","Type":"ContainerStarted","Data":"77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09"}
Jul 13 19:00:44.730713 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:44.730383    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:44.731853 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:44.731828    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" event={"ID":"6e489e9b-a37d-4a7f-a8f4-8f53153883c3","Type":"ContainerStarted","Data":"af005e63e712265ef9b73768a56c9b47186d61c604b26be5a1ef9d3ab895183e"}
Jul 13 19:00:44.732007 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:44.731959    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:44.732007 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:44.731985    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:44.733244 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:44.733197    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:00:44.755372 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:44.755329    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podStartSLOduration=1.88683093 podStartE2EDuration="19.755315108s" totalImagesPullingTime="17.868484178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:00:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:00:44.754644588 +0000 UTC m=+590.994944688" watchObservedRunningTime="2026-07-13 19:00:44.755315108 +0000 UTC m=+590.995615207"
Jul 13 19:00:44.771407 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:44.771349    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podStartSLOduration=2.001159215 podStartE2EDuration="20.771329555s" totalImagesPullingTime="18.77017034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:00:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:00:44.771322983 +0000 UTC m=+591.011623096" watchObservedRunningTime="2026-07-13 19:00:44.771329555 +0000 UTC m=+591.011629656"
Jul 13 19:00:45.736555 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:45.736511    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:00:45.737009 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:45.736587    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:45.737962 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:45.737937    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:00:46.740162 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:46.740122    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:00:50.741384 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:50.741349    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:00:50.741826 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:50.741798    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:00:51.745264 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:51.745232    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:00:51.745715 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:51.745683    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:00:54.323002 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:54.322970    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:00:54.323732 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:00:54.323709    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:01:00.742458 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:00.742416    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:01:01.746513 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:01.746470    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:01:10.742637 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:10.742591    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:01:11.746298 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:11.746256    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:01:20.741850 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:20.741762    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:01:21.746450 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:21.746409    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:01:30.742363 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:30.742332    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:01:31.746747 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:31.746712    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:01:44.353239 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:44.353178    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:48.357355 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:48.357311    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:58.502285 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.502248    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:58.620727 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.620689    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"]
Jul 13 19:01:58.620983 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.620929    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container" containerID="cri-o://5891c325999604fb984529e829aa8237a4d09c6b0f1ea4c2601878fee860b606" gracePeriod=30
Jul 13 19:01:58.621110 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.621013    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kube-rbac-proxy" containerID="cri-o://af005e63e712265ef9b73768a56c9b47186d61c604b26be5a1ef9d3ab895183e" gracePeriod=30
Jul 13 19:01:58.659309 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.659278    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:58.720190 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.720150    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"]
Jul 13 19:01:58.724087 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.724060    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.726700 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.726673    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-91c3d-predictor-serving-cert\""
Jul 13 19:01:58.726853 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.726787    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-91c3d-kube-rbac-proxy-sar-config\""
Jul 13 19:01:58.731130 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.731082    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"]
Jul 13 19:01:58.731581 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.731411    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container" containerID="cri-o://03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2" gracePeriod=30
Jul 13 19:01:58.731714 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.731613    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kube-rbac-proxy" containerID="cri-o://77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09" gracePeriod=30
Jul 13 19:01:58.733980 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.733857    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"]
Jul 13 19:01:58.738137 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.737923    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:58.791319 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.791282    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd92v\" (UniqueName: \"kubernetes.io/projected/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-api-access-sd92v\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.791319 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.791330    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ec0e05-4658-425d-87c8-379dbcd0769d-proxy-tls\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.791523 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.791356    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.802032 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.801989    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:58.805923 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.805895    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"]
Jul 13 19:01:58.809144 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.809121    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:58.811726 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.811702    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-91c3d-kube-rbac-proxy-sar-config\""
Jul 13 19:01:58.811858 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.811802    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-91c3d-predictor-serving-cert\""
Jul 13 19:01:58.820581 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.820553    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"]
Jul 13 19:01:58.892685 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.892645    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.892876 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.892702    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:58.892876 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.892744    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd92v\" (UniqueName: \"kubernetes.io/projected/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-api-access-sd92v\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.892876 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.892768    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:58.892876 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.892792    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ec0e05-4658-425d-87c8-379dbcd0769d-proxy-tls\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.892876 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.892822    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkgd\" (UniqueName: \"kubernetes.io/projected/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-api-access-5pkgd\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:58.893459 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.893439    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.894986 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.894953    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:58.895616 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.895592    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ec0e05-4658-425d-87c8-379dbcd0769d-proxy-tls\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.900984 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.900957    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd92v\" (UniqueName: \"kubernetes.io/projected/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-api-access-sd92v\") pod \"success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") " pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:58.994074 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.994027    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkgd\" (UniqueName: \"kubernetes.io/projected/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-api-access-5pkgd\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:58.994298 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.994102    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:58.994298 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.994156    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:58.994298 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:01:58.994291    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-serving-cert: secret "error-404-isvc-91c3d-predictor-serving-cert" not found
Jul 13 19:01:58.994479 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:01:58.994360    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls podName:5b77ffb2-b4b5-462f-ba67-092843462fb4 nodeName:}" failed. No retries permitted until 2026-07-13 19:01:59.494340109 +0000 UTC m=+665.734640193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls") pod "error-404-isvc-91c3d-predictor-84d6544c-dqr8h" (UID: "5b77ffb2-b4b5-462f-ba67-092843462fb4") : secret "error-404-isvc-91c3d-predictor-serving-cert" not found
Jul 13 19:01:58.994753 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:58.994729    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:59.005643 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.005612    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkgd\" (UniqueName: \"kubernetes.io/projected/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-api-access-5pkgd\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:59.008605 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.008578    2587 generic.go:356] "Generic (PLEG): container finished" podID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerID="77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09" exitCode=2
Jul 13 19:01:59.008766 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.008646    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" event={"ID":"9bf5f463-4192-4760-8fff-33b41a409c0e","Type":"ContainerDied","Data":"77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09"}
Jul 13 19:01:59.010579 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.010556    2587 generic.go:356] "Generic (PLEG): container finished" podID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerID="af005e63e712265ef9b73768a56c9b47186d61c604b26be5a1ef9d3ab895183e" exitCode=2
Jul 13 19:01:59.010675 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.010623    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" event={"ID":"6e489e9b-a37d-4a7f-a8f4-8f53153883c3","Type":"ContainerDied","Data":"af005e63e712265ef9b73768a56c9b47186d61c604b26be5a1ef9d3ab895183e"}
Jul 13 19:01:59.038210 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.038181    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:01:59.075218 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.075166    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:59.170616 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.170587    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"]
Jul 13 19:01:59.172477 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:01:59.172451    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ec0e05_4658_425d_87c8_379dbcd0769d.slice/crio-12c186a562646424200632365823b779a1ce811dda7702b72e806cc5613304f0 WatchSource:0}: Error finding container 12c186a562646424200632365823b779a1ce811dda7702b72e806cc5613304f0: Status 404 returned error can't find the container with id 12c186a562646424200632365823b779a1ce811dda7702b72e806cc5613304f0
Jul 13 19:01:59.498341 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.498251    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:59.500741 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.500716    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls\") pod \"error-404-isvc-91c3d-predictor-84d6544c-dqr8h\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") " pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:59.719431 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.719391    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:01:59.868546 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:01:59.868509    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"]
Jul 13 19:01:59.885620 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:01:59.885585    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b77ffb2_b4b5_462f_ba67_092843462fb4.slice/crio-d005e4e12a5241726450cd869997922dece4cbe15de265a001966ea88362b716 WatchSource:0}: Error finding container d005e4e12a5241726450cd869997922dece4cbe15de265a001966ea88362b716: Status 404 returned error can't find the container with id d005e4e12a5241726450cd869997922dece4cbe15de265a001966ea88362b716
Jul 13 19:02:00.015977 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.015940    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" event={"ID":"e2ec0e05-4658-425d-87c8-379dbcd0769d","Type":"ContainerStarted","Data":"f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd"}
Jul 13 19:02:00.016120 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.016015    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" event={"ID":"e2ec0e05-4658-425d-87c8-379dbcd0769d","Type":"ContainerStarted","Data":"879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732"}
Jul 13 19:02:00.016120 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.016047    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" event={"ID":"e2ec0e05-4658-425d-87c8-379dbcd0769d","Type":"ContainerStarted","Data":"12c186a562646424200632365823b779a1ce811dda7702b72e806cc5613304f0"}
Jul 13 19:02:00.016120 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.016083    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:02:00.016120 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.016111    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:02:00.017861 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.017800    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:02:00.018080 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.018059    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" event={"ID":"5b77ffb2-b4b5-462f-ba67-092843462fb4","Type":"ContainerStarted","Data":"5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37"}
Jul 13 19:02:00.018150 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.018109    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" event={"ID":"5b77ffb2-b4b5-462f-ba67-092843462fb4","Type":"ContainerStarted","Data":"d005e4e12a5241726450cd869997922dece4cbe15de265a001966ea88362b716"}
Jul 13 19:02:00.035188 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.035137    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podStartSLOduration=2.035117165 podStartE2EDuration="2.035117165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:02:00.033803407 +0000 UTC m=+666.274103509" watchObservedRunningTime="2026-07-13 19:02:00.035117165 +0000 UTC m=+666.275417270"
Jul 13 19:02:00.737441 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.737402    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 13 19:02:00.742756 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:00.742734    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:02:01.026088 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:01.026055    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" event={"ID":"5b77ffb2-b4b5-462f-ba67-092843462fb4","Type":"ContainerStarted","Data":"c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf"}
Jul 13 19:02:01.026368 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:01.026342    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:02:01.026436 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:01.026417    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:02:01.026585 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:01.026549    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:02:01.027783 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:01.027757    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:02:01.044773 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:01.044719    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podStartSLOduration=3.044703618 podStartE2EDuration="3.044703618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:02:01.044113444 +0000 UTC m=+667.284413544" watchObservedRunningTime="2026-07-13 19:02:01.044703618 +0000 UTC m=+667.285003717"
Jul 13 19:02:01.740474 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:01.740426    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 13 19:02:01.746037 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:01.746004    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:02:02.033092 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.033048    2587 generic.go:356] "Generic (PLEG): container finished" podID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerID="5891c325999604fb984529e829aa8237a4d09c6b0f1ea4c2601878fee860b606" exitCode=0
Jul 13 19:02:02.033287 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.033120    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" event={"ID":"6e489e9b-a37d-4a7f-a8f4-8f53153883c3","Type":"ContainerDied","Data":"5891c325999604fb984529e829aa8237a4d09c6b0f1ea4c2601878fee860b606"}
Jul 13 19:02:02.033450 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.033427    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:02:02.075063 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.075038    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:02:02.120280 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.120196    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwplp\" (UniqueName: \"kubernetes.io/projected/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-api-access-xwplp\") pod \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") "
Jul 13 19:02:02.120280 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.120264    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-proxy-tls\") pod \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") "
Jul 13 19:02:02.120462 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.120316    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-rbac-proxy-sar-config\") pod \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\" (UID: \"6e489e9b-a37d-4a7f-a8f4-8f53153883c3\") "
Jul 13 19:02:02.120762 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.120737    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e489e9b-a37d-4a7f-a8f4-8f53153883c3" (UID: "6e489e9b-a37d-4a7f-a8f4-8f53153883c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:02:02.122381 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.122356    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-api-access-xwplp" (OuterVolumeSpecName: "kube-api-access-xwplp") pod "6e489e9b-a37d-4a7f-a8f4-8f53153883c3" (UID: "6e489e9b-a37d-4a7f-a8f4-8f53153883c3"). InnerVolumeSpecName "kube-api-access-xwplp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:02:02.122499 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.122395    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e489e9b-a37d-4a7f-a8f4-8f53153883c3" (UID: "6e489e9b-a37d-4a7f-a8f4-8f53153883c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:02:02.221296 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.221254    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwplp\" (UniqueName: \"kubernetes.io/projected/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-api-access-xwplp\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:02:02.221296 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.221294    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:02:02.221476 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.221312    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e489e9b-a37d-4a7f-a8f4-8f53153883c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:02:02.267696 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.267666    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:02:02.322299 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.322181    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls\") pod \"9bf5f463-4192-4760-8fff-33b41a409c0e\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") "
Jul 13 19:02:02.322299 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.322286    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rz76\" (UniqueName: \"kubernetes.io/projected/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-api-access-5rz76\") pod \"9bf5f463-4192-4760-8fff-33b41a409c0e\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") "
Jul 13 19:02:02.322475 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.322326    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-rbac-proxy-sar-config\") pod \"9bf5f463-4192-4760-8fff-33b41a409c0e\" (UID: \"9bf5f463-4192-4760-8fff-33b41a409c0e\") "
Jul 13 19:02:02.322712 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.322688    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9bf5f463-4192-4760-8fff-33b41a409c0e" (UID: "9bf5f463-4192-4760-8fff-33b41a409c0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:02:02.324348 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.324324    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bf5f463-4192-4760-8fff-33b41a409c0e" (UID: "9bf5f463-4192-4760-8fff-33b41a409c0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:02:02.324448 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.324426    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-api-access-5rz76" (OuterVolumeSpecName: "kube-api-access-5rz76") pod "9bf5f463-4192-4760-8fff-33b41a409c0e" (UID: "9bf5f463-4192-4760-8fff-33b41a409c0e"). InnerVolumeSpecName "kube-api-access-5rz76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:02:02.423696 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.423658    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rz76\" (UniqueName: \"kubernetes.io/projected/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-api-access-5rz76\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:02:02.423696 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.423690    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf5f463-4192-4760-8fff-33b41a409c0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:02:02.423696 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:02.423701    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf5f463-4192-4760-8fff-33b41a409c0e-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:02:03.038955 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.038920    2587 generic.go:356] "Generic (PLEG): container finished" podID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerID="03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2" exitCode=0
Jul 13 19:02:03.039395 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.038992    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"
Jul 13 19:02:03.039395 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.039008    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" event={"ID":"9bf5f463-4192-4760-8fff-33b41a409c0e","Type":"ContainerDied","Data":"03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2"}
Jul 13 19:02:03.039395 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.039080    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9" event={"ID":"9bf5f463-4192-4760-8fff-33b41a409c0e","Type":"ContainerDied","Data":"c8fe03e561b0ab3ad888f72c19c726e86ed6c36e04ac0afae5a7306226638f19"}
Jul 13 19:02:03.039395 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.039121    2587 scope.go:117] "RemoveContainer" containerID="77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09"
Jul 13 19:02:03.040804 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.040781    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5" event={"ID":"6e489e9b-a37d-4a7f-a8f4-8f53153883c3","Type":"ContainerDied","Data":"b61e8dc0422b1873eecbf926eae420370c4445630cb726ac11aa309c10f42997"}
Jul 13 19:02:03.040938 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.040802    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"
Jul 13 19:02:03.047348 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.047332    2587 scope.go:117] "RemoveContainer" containerID="03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2"
Jul 13 19:02:03.055084 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.055060    2587 scope.go:117] "RemoveContainer" containerID="77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09"
Jul 13 19:02:03.055539 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:02:03.055518    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09\": container with ID starting with 77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09 not found: ID does not exist" containerID="77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09"
Jul 13 19:02:03.055628 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.055547    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09"} err="failed to get container status \"77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09\": rpc error: code = NotFound desc = could not find container \"77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09\": container with ID starting with 77622029b4d7ac87f5d382f79d44acfec71c0788aa3b2089c6cd9de6772dca09 not found: ID does not exist"
Jul 13 19:02:03.055628 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.055573    2587 scope.go:117] "RemoveContainer" containerID="03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2"
Jul 13 19:02:03.055846 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:02:03.055806    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2\": container with ID starting with 03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2 not found: ID does not exist" containerID="03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2"
Jul 13 19:02:03.055929 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.055846    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2"} err="failed to get container status \"03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2\": rpc error: code = NotFound desc = could not find container \"03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2\": container with ID starting with 03388d206e8f7c2d7d056b8e112a53f235798b704cad8f4a7ef5463724e4ccd2 not found: ID does not exist"
Jul 13 19:02:03.055929 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.055871    2587 scope.go:117] "RemoveContainer" containerID="af005e63e712265ef9b73768a56c9b47186d61c604b26be5a1ef9d3ab895183e"
Jul 13 19:02:03.056842 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.056819    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"]
Jul 13 19:02:03.061593 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.061574    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19366-predictor-6b66d49548-wrwt5"]
Jul 13 19:02:03.063926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.063906    2587 scope.go:117] "RemoveContainer" containerID="5891c325999604fb984529e829aa8237a4d09c6b0f1ea4c2601878fee860b606"
Jul 13 19:02:03.073373 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.073350    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"]
Jul 13 19:02:03.077711 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:03.077690    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19366-predictor-758f8cb57-swfg9"]
Jul 13 19:02:04.505332 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:04.505293    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" path="/var/lib/kubelet/pods/6e489e9b-a37d-4a7f-a8f4-8f53153883c3/volumes"
Jul 13 19:02:04.505911 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:04.505887    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" path="/var/lib/kubelet/pods/9bf5f463-4192-4760-8fff-33b41a409c0e/volumes"
Jul 13 19:02:06.031736 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:06.031709    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:02:06.032177 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:06.032153    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:02:07.038143 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:07.038112    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:02:07.038613 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:07.038587    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:02:16.033018 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:16.032971    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:02:17.039192 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:17.039153    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:02:26.032758 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:26.032712    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:02:27.038901 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:27.038860    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:02:34.397821 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:34.397785    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:35.216646 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:35.216609    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:36.032501 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:36.032459    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:02:37.039425 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:37.039385    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:02:44.524817 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.524784    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:44.663262 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.663226    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:44.679606 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679576    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"]
Jul 13 19:02:44.679904 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679890    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container"
Jul 13 19:02:44.679904 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679905    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679916    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kube-rbac-proxy"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679922    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kube-rbac-proxy"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679936    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679943    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679952    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kube-rbac-proxy"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.679957    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kube-rbac-proxy"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.680003    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kserve-container"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.680012    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e489e9b-a37d-4a7f-a8f4-8f53153883c3" containerName="kube-rbac-proxy"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.680018    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kube-rbac-proxy"
Jul 13 19:02:44.680081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.680024    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf5f463-4192-4760-8fff-33b41a409c0e" containerName="kserve-container"
Jul 13 19:02:44.683192 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.683170    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.685564 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.685542    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3a16a-predictor-serving-cert\""
Jul 13 19:02:44.685671 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.685565    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3a16a-kube-rbac-proxy-sar-config\""
Jul 13 19:02:44.695221 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.695167    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"]
Jul 13 19:02:44.774616 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.774580    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:44.780992 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.780927    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"]
Jul 13 19:02:44.784185 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.784164    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:44.786671 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.786649    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3a16a-kube-rbac-proxy-sar-config\""
Jul 13 19:02:44.786770 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.786650    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3a16a-predictor-serving-cert\""
Jul 13 19:02:44.795566 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.795541    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"]
Jul 13 19:02:44.863177 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.863137    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2947986d-5361-4e23-908d-c270c4637661-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.863177 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.863177    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qp28\" (UniqueName: \"kubernetes.io/projected/2947986d-5361-4e23-908d-c270c4637661-kube-api-access-2qp28\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.863465 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.863271    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2947986d-5361-4e23-908d-c270c4637661-proxy-tls\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.864140 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.864116    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:44.963934 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.963903    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2947986d-5361-4e23-908d-c270c4637661-proxy-tls\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.964127 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.963975    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb069095-a5f5-45bd-9451-6f3b145a1219-proxy-tls\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:44.964127 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.964008    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2947986d-5361-4e23-908d-c270c4637661-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.964127 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.964039    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxxq\" (UniqueName: \"kubernetes.io/projected/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-api-access-9mxxq\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:44.964127 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.964068    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qp28\" (UniqueName: \"kubernetes.io/projected/2947986d-5361-4e23-908d-c270c4637661-kube-api-access-2qp28\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.964381 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.964130    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:44.964700 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.964680    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2947986d-5361-4e23-908d-c270c4637661-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.966477 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.966455    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2947986d-5361-4e23-908d-c270c4637661-proxy-tls\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.971228 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.971192    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qp28\" (UniqueName: \"kubernetes.io/projected/2947986d-5361-4e23-908d-c270c4637661-kube-api-access-2qp28\") pod \"success-200-isvc-3a16a-predictor-68664fb5d6-5cmst\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") " pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:44.980351 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.980325    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:44.995269 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:44.995229    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:45.065524 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.065485    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb069095-a5f5-45bd-9451-6f3b145a1219-proxy-tls\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:45.065688 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.065540    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxxq\" (UniqueName: \"kubernetes.io/projected/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-api-access-9mxxq\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:45.065688 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.065588    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:45.066448 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.066422    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:45.068184 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.068159    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb069095-a5f5-45bd-9451-6f3b145a1219-proxy-tls\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:45.074136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.074109    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxxq\" (UniqueName: \"kubernetes.io/projected/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-api-access-9mxxq\") pod \"error-404-isvc-3a16a-predictor-59bd97769f-hwr4r\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") " pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:45.082749 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.082718    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:45.095016 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.094992    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:45.129419 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.129389    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"]
Jul 13 19:02:45.131994 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:02:45.131960    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2947986d_5361_4e23_908d_c270c4637661.slice/crio-6a467d2750b4cd4e7a498da0089cd57e936b7b06129ac97c742ae0d8278cbb89 WatchSource:0}: Error finding container 6a467d2750b4cd4e7a498da0089cd57e936b7b06129ac97c742ae0d8278cbb89: Status 404 returned error can't find the container with id 6a467d2750b4cd4e7a498da0089cd57e936b7b06129ac97c742ae0d8278cbb89
Jul 13 19:02:45.204659 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.204531    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" event={"ID":"2947986d-5361-4e23-908d-c270c4637661","Type":"ContainerStarted","Data":"6a467d2750b4cd4e7a498da0089cd57e936b7b06129ac97c742ae0d8278cbb89"}
Jul 13 19:02:45.229604 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:45.229579    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"]
Jul 13 19:02:45.246334 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:02:45.246310    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb069095_a5f5_45bd_9451_6f3b145a1219.slice/crio-16e891f287d77666d2b6fefdf573519c7240f9f322e4e61c3fd6431929c1b3f9 WatchSource:0}: Error finding container 16e891f287d77666d2b6fefdf573519c7240f9f322e4e61c3fd6431929c1b3f9: Status 404 returned error can't find the container with id 16e891f287d77666d2b6fefdf573519c7240f9f322e4e61c3fd6431929c1b3f9
Jul 13 19:02:46.032354 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.032271    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:02:46.211294 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.211255    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" event={"ID":"cb069095-a5f5-45bd-9451-6f3b145a1219","Type":"ContainerStarted","Data":"58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d"}
Jul 13 19:02:46.211488 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.211341    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:46.211488 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.211375    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:46.211488 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.211403    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" event={"ID":"cb069095-a5f5-45bd-9451-6f3b145a1219","Type":"ContainerStarted","Data":"9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563"}
Jul 13 19:02:46.211488 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.211433    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" event={"ID":"cb069095-a5f5-45bd-9451-6f3b145a1219","Type":"ContainerStarted","Data":"16e891f287d77666d2b6fefdf573519c7240f9f322e4e61c3fd6431929c1b3f9"}
Jul 13 19:02:46.212875 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.212845    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:02:46.213172 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.213148    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" event={"ID":"2947986d-5361-4e23-908d-c270c4637661","Type":"ContainerStarted","Data":"328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f"}
Jul 13 19:02:46.213309 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.213211    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:46.213309 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.213233    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:46.213309 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.213251    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" event={"ID":"2947986d-5361-4e23-908d-c270c4637661","Type":"ContainerStarted","Data":"5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e"}
Jul 13 19:02:46.214171 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.214147    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 19:02:46.233384 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.233338    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podStartSLOduration=2.233323351 podStartE2EDuration="2.233323351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:02:46.232248072 +0000 UTC m=+712.472548163" watchObservedRunningTime="2026-07-13 19:02:46.233323351 +0000 UTC m=+712.473623451"
Jul 13 19:02:46.249714 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:46.249662    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podStartSLOduration=2.249645233 podStartE2EDuration="2.249645233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:02:46.24845346 +0000 UTC m=+712.488753597" watchObservedRunningTime="2026-07-13 19:02:46.249645233 +0000 UTC m=+712.489945334"
Jul 13 19:02:47.039838 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:47.039803    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:02:47.216988 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:47.216948    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 19:02:47.217180 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:47.217034    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:02:52.221563 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:52.221533    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:02:52.222017 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:52.221760    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:02:52.222017 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:52.221982    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 19:02:52.222410 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:52.222382    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:02:56.033364 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:02:56.033329    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:03:02.221934 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:02.221894    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 19:03:02.222421 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:02.222387    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:03:08.767166 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:08.767127    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:09.490715 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:09.490675    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:12.222537 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:12.222499    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:03:12.222921 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:12.222501    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 19:03:22.222668 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:22.222626    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 19:03:22.223131 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:22.222628    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:03:32.223057 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:32.223024    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:03:32.223533 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:32.223220    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:03:44.737779 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:44.737736    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:45.656601 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:03:45.656567    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:36.161600 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:05:36.161560    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:36.220591 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:05:36.220539    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:54.349081 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:05:54.348992    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:05:54.349554 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:05:54.349327    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:07:03.180470 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:07:03.180428    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:07:25.077802 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:07:25.077723    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:07:37.676603 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:07:37.676563    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:10:54.375523 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:10:54.375489    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:10:54.376472 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:10:54.376446    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:11:23.502033 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.501999    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:23.611045 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.611015    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"]
Jul 13 19:11:23.611294 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.611254    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" containerID="cri-o://879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732" gracePeriod=30
Jul 13 19:11:23.611374 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.611333    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kube-rbac-proxy" containerID="cri-o://f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd" gracePeriod=30
Jul 13 19:11:23.677332 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.677300    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"]
Jul 13 19:11:23.680989 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.680965    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.683829 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.683800    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-68f71-predictor-serving-cert\""
Jul 13 19:11:23.683979 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.683831    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-68f71-kube-rbac-proxy-sar-config\""
Jul 13 19:11:23.697452 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.697417    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"]
Jul 13 19:11:23.706887 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.706858    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.707017 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.706899    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.707075 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.707033    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55mlc\" (UniqueName: \"kubernetes.io/projected/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-api-access-55mlc\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.754250 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.754161    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"]
Jul 13 19:11:23.757423 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.757405    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.760374 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.760349    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-68f71-predictor-serving-cert\""
Jul 13 19:11:23.760657 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.760640    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-68f71-kube-rbac-proxy-sar-config\""
Jul 13 19:11:23.769406 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.769374    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"]
Jul 13 19:11:23.807894 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.807863    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5fm\" (UniqueName: \"kubernetes.io/projected/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-api-access-rl5fm\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.808058 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.807906    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55mlc\" (UniqueName: \"kubernetes.io/projected/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-api-access-55mlc\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.808058 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.807927    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.808058 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:11:23.808019    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-68f71-predictor-serving-cert: secret "success-200-isvc-68f71-predictor-serving-cert" not found
Jul 13 19:11:23.808247 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:11:23.808080    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls podName:8692dd78-4ecc-4590-80b2-f69841a4e023 nodeName:}" failed. No retries permitted until 2026-07-13 19:11:24.308060428 +0000 UTC m=+1230.548360506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls") pod "success-200-isvc-68f71-predictor-dc77967f4-s6d9d" (UID: "8692dd78-4ecc-4590-80b2-f69841a4e023") : secret "success-200-isvc-68f71-predictor-serving-cert" not found
Jul 13 19:11:23.808247 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.808095    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.808247 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.808120    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7377b88d-c036-4ff7-89de-fab1bc293e1a-proxy-tls\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.808247 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.808163    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.808777 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.808754    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.821899 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.821866    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55mlc\" (UniqueName: \"kubernetes.io/projected/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-api-access-55mlc\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:23.832126 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.832096    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"]
Jul 13 19:11:23.832364 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.832340    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container" containerID="cri-o://5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37" gracePeriod=30
Jul 13 19:11:23.832454 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.832365    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kube-rbac-proxy" containerID="cri-o://c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf" gracePeriod=30
Jul 13 19:11:23.908836 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.908796    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.909027 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.908870    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5fm\" (UniqueName: \"kubernetes.io/projected/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-api-access-rl5fm\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.909027 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.908924    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7377b88d-c036-4ff7-89de-fab1bc293e1a-proxy-tls\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.909645 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.909617    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.911515 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.911494    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7377b88d-c036-4ff7-89de-fab1bc293e1a-proxy-tls\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:23.922308 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:23.922275    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5fm\" (UniqueName: \"kubernetes.io/projected/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-api-access-rl5fm\") pod \"error-404-isvc-68f71-predictor-6478fb4ccf-w9blh\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:24.067468 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.067425    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:24.209239 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:11:24.209176    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7377b88d_c036_4ff7_89de_fab1bc293e1a.slice/crio-fd89e31aebbb88712d7638f4ca76b8c7022350084aac0ae0cc7c6ade62266bff WatchSource:0}: Error finding container fd89e31aebbb88712d7638f4ca76b8c7022350084aac0ae0cc7c6ade62266bff: Status 404 returned error can't find the container with id fd89e31aebbb88712d7638f4ca76b8c7022350084aac0ae0cc7c6ade62266bff
Jul 13 19:11:24.209560 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.209534    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"]
Jul 13 19:11:24.226233 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.226189    2587 generic.go:356] "Generic (PLEG): container finished" podID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerID="c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf" exitCode=2
Jul 13 19:11:24.226334 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.226245    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" event={"ID":"5b77ffb2-b4b5-462f-ba67-092843462fb4","Type":"ContainerDied","Data":"c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf"}
Jul 13 19:11:24.227665 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.227641    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" event={"ID":"7377b88d-c036-4ff7-89de-fab1bc293e1a","Type":"ContainerStarted","Data":"fd89e31aebbb88712d7638f4ca76b8c7022350084aac0ae0cc7c6ade62266bff"}
Jul 13 19:11:24.229564 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.229537    2587 generic.go:356] "Generic (PLEG): container finished" podID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerID="f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd" exitCode=2
Jul 13 19:11:24.229649 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.229578    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" event={"ID":"e2ec0e05-4658-425d-87c8-379dbcd0769d","Type":"ContainerDied","Data":"f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd"}
Jul 13 19:11:24.312102 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.312073    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:24.314389 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.314365    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls\") pod \"success-200-isvc-68f71-predictor-dc77967f4-s6d9d\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") " pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:24.592071 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.592029    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:24.735850 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.735504    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"]
Jul 13 19:11:24.738182 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:11:24.738156    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8692dd78_4ecc_4590_80b2_f69841a4e023.slice/crio-d7d7c0a84a7a64ed9ff1e117af82bfd11358b7b9c25f9bae0af22430214fed07 WatchSource:0}: Error finding container d7d7c0a84a7a64ed9ff1e117af82bfd11358b7b9c25f9bae0af22430214fed07: Status 404 returned error can't find the container with id d7d7c0a84a7a64ed9ff1e117af82bfd11358b7b9c25f9bae0af22430214fed07
Jul 13 19:11:24.816926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.816883    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:24.914285 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:24.914249    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:25.235360 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.235256    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" event={"ID":"7377b88d-c036-4ff7-89de-fab1bc293e1a","Type":"ContainerStarted","Data":"aec3b254fb88818db1906ffb54bc539a54145b42550e785b1027b45e88774561"}
Jul 13 19:11:25.235360 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.235330    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" event={"ID":"7377b88d-c036-4ff7-89de-fab1bc293e1a","Type":"ContainerStarted","Data":"9500e6d35342f77b21a10426f0e03ee27eb23966d34a3af5bbf91ddd62d9fec1"}
Jul 13 19:11:25.235360 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.235363    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:25.235620 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.235383    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:25.236955 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.236922    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:11:25.237252 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.237225    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" event={"ID":"8692dd78-4ecc-4590-80b2-f69841a4e023","Type":"ContainerStarted","Data":"1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd"}
Jul 13 19:11:25.237353 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.237276    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" event={"ID":"8692dd78-4ecc-4590-80b2-f69841a4e023","Type":"ContainerStarted","Data":"9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1"}
Jul 13 19:11:25.237353 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.237305    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" event={"ID":"8692dd78-4ecc-4590-80b2-f69841a4e023","Type":"ContainerStarted","Data":"d7d7c0a84a7a64ed9ff1e117af82bfd11358b7b9c25f9bae0af22430214fed07"}
Jul 13 19:11:25.237353 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.237330    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:25.261231 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.261161    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podStartSLOduration=2.261141959 podStartE2EDuration="2.261141959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:11:25.258493974 +0000 UTC m=+1231.498794074" watchObservedRunningTime="2026-07-13 19:11:25.261141959 +0000 UTC m=+1231.501442059"
Jul 13 19:11:25.286316 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:25.286260    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podStartSLOduration=2.2862396179999998 podStartE2EDuration="2.286239618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:11:25.283012469 +0000 UTC m=+1231.523312571" watchObservedRunningTime="2026-07-13 19:11:25.286239618 +0000 UTC m=+1231.526539719"
Jul 13 19:11:26.026664 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:26.026622    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 13 19:11:26.032689 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:26.032652    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:11:26.241023 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:26.240984    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:11:26.241253 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:26.241053    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:26.242287 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:26.242262    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:11:27.033765 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.033716    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 13 19:11:27.038810 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.038780    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:11:27.068636 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.068613    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:11:27.137071 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.137038    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-rbac-proxy-sar-config\") pod \"e2ec0e05-4658-425d-87c8-379dbcd0769d\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") "
Jul 13 19:11:27.137259 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.137079    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ec0e05-4658-425d-87c8-379dbcd0769d-proxy-tls\") pod \"e2ec0e05-4658-425d-87c8-379dbcd0769d\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") "
Jul 13 19:11:27.137259 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.137115    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd92v\" (UniqueName: \"kubernetes.io/projected/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-api-access-sd92v\") pod \"e2ec0e05-4658-425d-87c8-379dbcd0769d\" (UID: \"e2ec0e05-4658-425d-87c8-379dbcd0769d\") "
Jul 13 19:11:27.137483 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.137462    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2ec0e05-4658-425d-87c8-379dbcd0769d" (UID: "e2ec0e05-4658-425d-87c8-379dbcd0769d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:11:27.139328 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.139302    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-api-access-sd92v" (OuterVolumeSpecName: "kube-api-access-sd92v") pod "e2ec0e05-4658-425d-87c8-379dbcd0769d" (UID: "e2ec0e05-4658-425d-87c8-379dbcd0769d"). InnerVolumeSpecName "kube-api-access-sd92v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:11:27.139328 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.139315    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ec0e05-4658-425d-87c8-379dbcd0769d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2ec0e05-4658-425d-87c8-379dbcd0769d" (UID: "e2ec0e05-4658-425d-87c8-379dbcd0769d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:11:27.238074 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.238040    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ec0e05-4658-425d-87c8-379dbcd0769d-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:11:27.238074 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.238067    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd92v\" (UniqueName: \"kubernetes.io/projected/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-api-access-sd92v\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:11:27.238074 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.238078    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ec0e05-4658-425d-87c8-379dbcd0769d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:11:27.246644 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.246612    2587 generic.go:356] "Generic (PLEG): container finished" podID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerID="879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732" exitCode=0
Jul 13 19:11:27.246780 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.246657    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"
Jul 13 19:11:27.246780 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.246693    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" event={"ID":"e2ec0e05-4658-425d-87c8-379dbcd0769d","Type":"ContainerDied","Data":"879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732"}
Jul 13 19:11:27.246780 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.246759    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4" event={"ID":"e2ec0e05-4658-425d-87c8-379dbcd0769d","Type":"ContainerDied","Data":"12c186a562646424200632365823b779a1ce811dda7702b72e806cc5613304f0"}
Jul 13 19:11:27.246889 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.246787    2587 scope.go:117] "RemoveContainer" containerID="f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd"
Jul 13 19:11:27.247392 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.247355    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:11:27.256460 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.256437    2587 scope.go:117] "RemoveContainer" containerID="879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732"
Jul 13 19:11:27.264297 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.264276    2587 scope.go:117] "RemoveContainer" containerID="f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd"
Jul 13 19:11:27.264591 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:11:27.264573    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd\": container with ID starting with f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd not found: ID does not exist" containerID="f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd"
Jul 13 19:11:27.264666 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.264604    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd"} err="failed to get container status \"f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd\": rpc error: code = NotFound desc = could not find container \"f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd\": container with ID starting with f67e8b2f67a0a732e108916f57b0281b0fa632f57418d5ebc4fc7f05971e24cd not found: ID does not exist"
Jul 13 19:11:27.264666 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.264627    2587 scope.go:117] "RemoveContainer" containerID="879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732"
Jul 13 19:11:27.264850 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:11:27.264832    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732\": container with ID starting with 879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732 not found: ID does not exist" containerID="879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732"
Jul 13 19:11:27.264902 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.264859    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732"} err="failed to get container status \"879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732\": rpc error: code = NotFound desc = could not find container \"879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732\": container with ID starting with 879bed264b36096f313561351f326e5ade2fe2d512edb5edd17c1092aeae8732 not found: ID does not exist"
Jul 13 19:11:27.269967 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.269943    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"]
Jul 13 19:11:27.274775 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.274751    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91c3d-predictor-cc8f58fd-kbsz4"]
Jul 13 19:11:27.379084 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.379059    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:11:27.439896 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.439865    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls\") pod \"5b77ffb2-b4b5-462f-ba67-092843462fb4\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") "
Jul 13 19:11:27.440067 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.439904    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pkgd\" (UniqueName: \"kubernetes.io/projected/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-api-access-5pkgd\") pod \"5b77ffb2-b4b5-462f-ba67-092843462fb4\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") "
Jul 13 19:11:27.440067 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.439937    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-rbac-proxy-sar-config\") pod \"5b77ffb2-b4b5-462f-ba67-092843462fb4\" (UID: \"5b77ffb2-b4b5-462f-ba67-092843462fb4\") "
Jul 13 19:11:27.440373 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.440350    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b77ffb2-b4b5-462f-ba67-092843462fb4" (UID: "5b77ffb2-b4b5-462f-ba67-092843462fb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:11:27.442047 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.442026    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b77ffb2-b4b5-462f-ba67-092843462fb4" (UID: "5b77ffb2-b4b5-462f-ba67-092843462fb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:11:27.442103 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.442085    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-api-access-5pkgd" (OuterVolumeSpecName: "kube-api-access-5pkgd") pod "5b77ffb2-b4b5-462f-ba67-092843462fb4" (UID: "5b77ffb2-b4b5-462f-ba67-092843462fb4"). InnerVolumeSpecName "kube-api-access-5pkgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:11:27.541383 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.541350    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:11:27.541383 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.541380    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b77ffb2-b4b5-462f-ba67-092843462fb4-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:11:27.541383 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:27.541390    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkgd\" (UniqueName: \"kubernetes.io/projected/5b77ffb2-b4b5-462f-ba67-092843462fb4-kube-api-access-5pkgd\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:11:28.252987 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.252902    2587 generic.go:356] "Generic (PLEG): container finished" podID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerID="5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37" exitCode=0
Jul 13 19:11:28.252987 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.252951    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" event={"ID":"5b77ffb2-b4b5-462f-ba67-092843462fb4","Type":"ContainerDied","Data":"5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37"}
Jul 13 19:11:28.252987 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.252971    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"
Jul 13 19:11:28.253472 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.253040    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h" event={"ID":"5b77ffb2-b4b5-462f-ba67-092843462fb4","Type":"ContainerDied","Data":"d005e4e12a5241726450cd869997922dece4cbe15de265a001966ea88362b716"}
Jul 13 19:11:28.253472 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.253081    2587 scope.go:117] "RemoveContainer" containerID="c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf"
Jul 13 19:11:28.262153 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.262133    2587 scope.go:117] "RemoveContainer" containerID="5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37"
Jul 13 19:11:28.269500 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.269482    2587 scope.go:117] "RemoveContainer" containerID="c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf"
Jul 13 19:11:28.269757 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:11:28.269739    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf\": container with ID starting with c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf not found: ID does not exist" containerID="c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf"
Jul 13 19:11:28.269820 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.269763    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf"} err="failed to get container status \"c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf\": rpc error: code = NotFound desc = could not find container \"c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf\": container with ID starting with c4488bcf91740b04367b91ee69e06706fd8b1cd700ac0fff2601aaa809c402bf not found: ID does not exist"
Jul 13 19:11:28.269820 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.269780    2587 scope.go:117] "RemoveContainer" containerID="5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37"
Jul 13 19:11:28.269992 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:11:28.269973    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37\": container with ID starting with 5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37 not found: ID does not exist" containerID="5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37"
Jul 13 19:11:28.270028 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.269997    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37"} err="failed to get container status \"5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37\": rpc error: code = NotFound desc = could not find container \"5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37\": container with ID starting with 5da2649b4d70b8e0f6c70821cc92db1353a15d6c49e65dbbfc3bd7cc34719f37 not found: ID does not exist"
Jul 13 19:11:28.283168 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.283139    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"]
Jul 13 19:11:28.290915 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.290890    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91c3d-predictor-84d6544c-dqr8h"]
Jul 13 19:11:28.514467 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.514383    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" path="/var/lib/kubelet/pods/5b77ffb2-b4b5-462f-ba67-092843462fb4/volumes"
Jul 13 19:11:28.514972 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:28.514954    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" path="/var/lib/kubelet/pods/e2ec0e05-4658-425d-87c8-379dbcd0769d/volumes"
Jul 13 19:11:31.245718 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:31.245688    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:11:31.246275 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:31.246248    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:11:32.251675 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:32.251644    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:11:32.252173 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:32.252143    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:11:41.247267 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:41.247226    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:11:42.252622 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:42.252580    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:11:51.246847 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:51.246750    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:11:52.252231 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:52.252173    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:11:59.483545 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.483504    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:59.601497 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.601464    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"]
Jul 13 19:11:59.601707 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.601675    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container" containerID="cri-o://328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f" gracePeriod=30
Jul 13 19:11:59.601863 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.601705    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kube-rbac-proxy" containerID="cri-o://5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e" gracePeriod=30
Jul 13 19:11:59.605551 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.605521    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:59.676020 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.675979    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"]
Jul 13 19:11:59.676306 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.676270    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container" containerID="cri-o://9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563" gracePeriod=30
Jul 13 19:11:59.676442 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.676341    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kube-rbac-proxy" containerID="cri-o://58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d" gracePeriod=30
Jul 13 19:11:59.752427 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752352    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"]
Jul 13 19:11:59.752733 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752719    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kube-rbac-proxy"
Jul 13 19:11:59.752775 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752735    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kube-rbac-proxy"
Jul 13 19:11:59.752775 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752751    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container"
Jul 13 19:11:59.752775 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752756    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container"
Jul 13 19:11:59.752775 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752768    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kube-rbac-proxy"
Jul 13 19:11:59.752775 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752774    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kube-rbac-proxy"
Jul 13 19:11:59.752926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752782    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container"
Jul 13 19:11:59.752926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752789    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container"
Jul 13 19:11:59.752926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752847    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kserve-container"
Jul 13 19:11:59.752926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752856    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kube-rbac-proxy"
Jul 13 19:11:59.752926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752863    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b77ffb2-b4b5-462f-ba67-092843462fb4" containerName="kserve-container"
Jul 13 19:11:59.752926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.752871    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ec0e05-4658-425d-87c8-379dbcd0769d" containerName="kube-rbac-proxy"
Jul 13 19:11:59.755949 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.755932    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:11:59.762333 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.762298    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e56dd-kube-rbac-proxy-sar-config\""
Jul 13 19:11:59.762333 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.762321    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e56dd-predictor-serving-cert\""
Jul 13 19:11:59.772417 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.772389    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"]
Jul 13 19:11:59.901108 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.901073    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:11:59.901348 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.901118    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7lp\" (UniqueName: \"kubernetes.io/projected/686d93a4-2a33-4158-be50-5531c78df11f-kube-api-access-4r7lp\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:11:59.901348 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.901286    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/686d93a4-2a33-4158-be50-5531c78df11f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:11:59.903179 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.903149    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"]
Jul 13 19:11:59.906833 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.906811    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:11:59.915369 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.915349    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e56dd-predictor-serving-cert\""
Jul 13 19:11:59.915478 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.915349    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e56dd-kube-rbac-proxy-sar-config\""
Jul 13 19:11:59.933816 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:11:59.933782    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"]
Jul 13 19:12:00.002257 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.002224    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/686d93a4-2a33-4158-be50-5531c78df11f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:00.002491 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.002265    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mwt9\" (UniqueName: \"kubernetes.io/projected/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-api-access-6mwt9\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.002491 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.002294    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:00.002491 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.002318    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7lp\" (UniqueName: \"kubernetes.io/projected/686d93a4-2a33-4158-be50-5531c78df11f-kube-api-access-4r7lp\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:00.002491 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.002367    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e5a392-9660-43eb-8494-ee0c8eb2702e-proxy-tls\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.002491 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:00.002432    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-serving-cert: secret "success-200-isvc-e56dd-predictor-serving-cert" not found
Jul 13 19:12:00.002491 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:00.002493    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls podName:686d93a4-2a33-4158-be50-5531c78df11f nodeName:}" failed. No retries permitted until 2026-07-13 19:12:00.50247449 +0000 UTC m=+1266.742774583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls") pod "success-200-isvc-e56dd-predictor-6b959756d6-cslzx" (UID: "686d93a4-2a33-4158-be50-5531c78df11f") : secret "success-200-isvc-e56dd-predictor-serving-cert" not found
Jul 13 19:12:00.002815 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.002687    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.002989 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.002966    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/686d93a4-2a33-4158-be50-5531c78df11f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:00.016307 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.016279    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7lp\" (UniqueName: \"kubernetes.io/projected/686d93a4-2a33-4158-be50-5531c78df11f-kube-api-access-4r7lp\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:00.103712 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.103672    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e5a392-9660-43eb-8494-ee0c8eb2702e-proxy-tls\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.103916 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.103732    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.103916 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.103810    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mwt9\" (UniqueName: \"kubernetes.io/projected/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-api-access-6mwt9\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.104551 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.104520    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.106675 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.106646    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e5a392-9660-43eb-8494-ee0c8eb2702e-proxy-tls\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.119725 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.119701    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mwt9\" (UniqueName: \"kubernetes.io/projected/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-api-access-6mwt9\") pod \"error-404-isvc-e56dd-predictor-7f8878c579-l8lf8\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") " pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.216999 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.216958    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:00.361785 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.361759    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"]
Jul 13 19:12:00.363020 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:12:00.362993    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e5a392_9660_43eb_8494_ee0c8eb2702e.slice/crio-fbed37baea00dc4df91e39c55dec0cea3d0659cfa1533b0fa6bb2d013d9320cd WatchSource:0}: Error finding container fbed37baea00dc4df91e39c55dec0cea3d0659cfa1533b0fa6bb2d013d9320cd: Status 404 returned error can't find the container with id fbed37baea00dc4df91e39c55dec0cea3d0659cfa1533b0fa6bb2d013d9320cd
Jul 13 19:12:00.382450 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.382416    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" event={"ID":"91e5a392-9660-43eb-8494-ee0c8eb2702e","Type":"ContainerStarted","Data":"fbed37baea00dc4df91e39c55dec0cea3d0659cfa1533b0fa6bb2d013d9320cd"}
Jul 13 19:12:00.384226 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.384184    2587 generic.go:356] "Generic (PLEG): container finished" podID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerID="58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d" exitCode=2
Jul 13 19:12:00.384333 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.384239    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" event={"ID":"cb069095-a5f5-45bd-9451-6f3b145a1219","Type":"ContainerDied","Data":"58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d"}
Jul 13 19:12:00.385831 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.385809    2587 generic.go:356] "Generic (PLEG): container finished" podID="2947986d-5361-4e23-908d-c270c4637661" containerID="5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e" exitCode=2
Jul 13 19:12:00.385907 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.385855    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" event={"ID":"2947986d-5361-4e23-908d-c270c4637661","Type":"ContainerDied","Data":"5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e"}
Jul 13 19:12:00.508004 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.507957    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:00.511456 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.511425    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls\") pod \"success-200-isvc-e56dd-predictor-6b959756d6-cslzx\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") " pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:00.671770 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.671670    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:00.822471 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:00.822440    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"]
Jul 13 19:12:00.826850 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:12:00.826825    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686d93a4_2a33_4158_be50_5531c78df11f.slice/crio-a5b1e250cac0548908c3ce310ef4118cb3cb8c32f4d59bc6fb0576ea7d4d890a WatchSource:0}: Error finding container a5b1e250cac0548908c3ce310ef4118cb3cb8c32f4d59bc6fb0576ea7d4d890a: Status 404 returned error can't find the container with id a5b1e250cac0548908c3ce310ef4118cb3cb8c32f4d59bc6fb0576ea7d4d890a
Jul 13 19:12:01.247146 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.247058    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:12:01.392428 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.392391    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" event={"ID":"91e5a392-9660-43eb-8494-ee0c8eb2702e","Type":"ContainerStarted","Data":"70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4"}
Jul 13 19:12:01.392620 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.392462    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" event={"ID":"91e5a392-9660-43eb-8494-ee0c8eb2702e","Type":"ContainerStarted","Data":"2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e"}
Jul 13 19:12:01.392620 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.392496    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:01.392620 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.392537    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:01.394151 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.394120    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 19:12:01.394505 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.394481    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" event={"ID":"686d93a4-2a33-4158-be50-5531c78df11f","Type":"ContainerStarted","Data":"70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f"}
Jul 13 19:12:01.394611 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.394535    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:01.394611 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.394560    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" event={"ID":"686d93a4-2a33-4158-be50-5531c78df11f","Type":"ContainerStarted","Data":"a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e"}
Jul 13 19:12:01.394611 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.394582    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:01.394611 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.394599    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" event={"ID":"686d93a4-2a33-4158-be50-5531c78df11f","Type":"ContainerStarted","Data":"a5b1e250cac0548908c3ce310ef4118cb3cb8c32f4d59bc6fb0576ea7d4d890a"}
Jul 13 19:12:01.395610 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.395587    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 19:12:01.463835 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.463779    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podStartSLOduration=2.463757192 podStartE2EDuration="2.463757192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:12:01.424054896 +0000 UTC m=+1267.664355004" watchObservedRunningTime="2026-07-13 19:12:01.463757192 +0000 UTC m=+1267.704057293"
Jul 13 19:12:01.464318 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.464290    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podStartSLOduration=2.464278283 podStartE2EDuration="2.464278283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:12:01.462919728 +0000 UTC m=+1267.703219827" watchObservedRunningTime="2026-07-13 19:12:01.464278283 +0000 UTC m=+1267.704578382"
Jul 13 19:12:01.522060 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.522029    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:01.617108 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:01.617075    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:02.217650 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:02.217608    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 13 19:12:02.217825 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:02.217615    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 13 19:12:02.222101 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:02.222072    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 19:12:02.223339 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:02.223317    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:12:02.252344 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:02.252307    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:12:02.399430 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:02.399391    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 19:12:02.399615 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:02.399463    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 19:12:03.360423 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.360395    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:12:03.408049 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.407524    2587 generic.go:356] "Generic (PLEG): container finished" podID="2947986d-5361-4e23-908d-c270c4637661" containerID="328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f" exitCode=0
Jul 13 19:12:03.408049 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.407584    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"
Jul 13 19:12:03.408049 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.407684    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" event={"ID":"2947986d-5361-4e23-908d-c270c4637661","Type":"ContainerDied","Data":"328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f"}
Jul 13 19:12:03.408049 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.407757    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst" event={"ID":"2947986d-5361-4e23-908d-c270c4637661","Type":"ContainerDied","Data":"6a467d2750b4cd4e7a498da0089cd57e936b7b06129ac97c742ae0d8278cbb89"}
Jul 13 19:12:03.408049 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.407802    2587 scope.go:117] "RemoveContainer" containerID="5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e"
Jul 13 19:12:03.420698 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.420676    2587 scope.go:117] "RemoveContainer" containerID="328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f"
Jul 13 19:12:03.428527 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.428505    2587 scope.go:117] "RemoveContainer" containerID="5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e"
Jul 13 19:12:03.428852 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:03.428830    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e\": container with ID starting with 5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e not found: ID does not exist" containerID="5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e"
Jul 13 19:12:03.428930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.428858    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e"} err="failed to get container status \"5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e\": rpc error: code = NotFound desc = could not find container \"5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e\": container with ID starting with 5f51f20aab339ede51275f50c2d85fa40c2ac79de46f6507a6ea14d47ca6188e not found: ID does not exist"
Jul 13 19:12:03.428930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.428879    2587 scope.go:117] "RemoveContainer" containerID="328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f"
Jul 13 19:12:03.429105 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:03.429090    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f\": container with ID starting with 328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f not found: ID does not exist" containerID="328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f"
Jul 13 19:12:03.429160 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.429113    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f"} err="failed to get container status \"328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f\": rpc error: code = NotFound desc = could not find container \"328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f\": container with ID starting with 328bcf2b7e0518075ac84c16490d901e89ac5c5c110e03db742c1fc36b2ecb1f not found: ID does not exist"
Jul 13 19:12:03.432410 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.432390    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2947986d-5361-4e23-908d-c270c4637661-proxy-tls\") pod \"2947986d-5361-4e23-908d-c270c4637661\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") "
Jul 13 19:12:03.432479 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.432425    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qp28\" (UniqueName: \"kubernetes.io/projected/2947986d-5361-4e23-908d-c270c4637661-kube-api-access-2qp28\") pod \"2947986d-5361-4e23-908d-c270c4637661\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") "
Jul 13 19:12:03.432479 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.432462    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2947986d-5361-4e23-908d-c270c4637661-kube-rbac-proxy-sar-config\") pod \"2947986d-5361-4e23-908d-c270c4637661\" (UID: \"2947986d-5361-4e23-908d-c270c4637661\") "
Jul 13 19:12:03.432899 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.432875    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2947986d-5361-4e23-908d-c270c4637661-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2947986d-5361-4e23-908d-c270c4637661" (UID: "2947986d-5361-4e23-908d-c270c4637661"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:12:03.434522 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.434495    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2947986d-5361-4e23-908d-c270c4637661-kube-api-access-2qp28" (OuterVolumeSpecName: "kube-api-access-2qp28") pod "2947986d-5361-4e23-908d-c270c4637661" (UID: "2947986d-5361-4e23-908d-c270c4637661"). InnerVolumeSpecName "kube-api-access-2qp28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:12:03.434593 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.434528    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2947986d-5361-4e23-908d-c270c4637661-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2947986d-5361-4e23-908d-c270c4637661" (UID: "2947986d-5361-4e23-908d-c270c4637661"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:12:03.533302 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.533263    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2947986d-5361-4e23-908d-c270c4637661-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:03.533302 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.533295    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qp28\" (UniqueName: \"kubernetes.io/projected/2947986d-5361-4e23-908d-c270c4637661-kube-api-access-2qp28\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:03.533302 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.533305    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2947986d-5361-4e23-908d-c270c4637661-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:03.747524 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.747491    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"]
Jul 13 19:12:03.755687 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.755655    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a16a-predictor-68664fb5d6-5cmst"]
Jul 13 19:12:03.833596 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.833568    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:12:03.936321 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.936278    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mxxq\" (UniqueName: \"kubernetes.io/projected/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-api-access-9mxxq\") pod \"cb069095-a5f5-45bd-9451-6f3b145a1219\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") "
Jul 13 19:12:03.936534 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.936362    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-rbac-proxy-sar-config\") pod \"cb069095-a5f5-45bd-9451-6f3b145a1219\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") "
Jul 13 19:12:03.936534 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.936509    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb069095-a5f5-45bd-9451-6f3b145a1219-proxy-tls\") pod \"cb069095-a5f5-45bd-9451-6f3b145a1219\" (UID: \"cb069095-a5f5-45bd-9451-6f3b145a1219\") "
Jul 13 19:12:03.936713 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.936691    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb069095-a5f5-45bd-9451-6f3b145a1219" (UID: "cb069095-a5f5-45bd-9451-6f3b145a1219"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:12:03.936804 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.936789    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:03.938530 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.938478    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb069095-a5f5-45bd-9451-6f3b145a1219-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb069095-a5f5-45bd-9451-6f3b145a1219" (UID: "cb069095-a5f5-45bd-9451-6f3b145a1219"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:12:03.938530 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:03.938512    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-api-access-9mxxq" (OuterVolumeSpecName: "kube-api-access-9mxxq") pod "cb069095-a5f5-45bd-9451-6f3b145a1219" (UID: "cb069095-a5f5-45bd-9451-6f3b145a1219"). InnerVolumeSpecName "kube-api-access-9mxxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:12:04.037347 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.037308    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb069095-a5f5-45bd-9451-6f3b145a1219-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:04.037347 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.037337    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mxxq\" (UniqueName: \"kubernetes.io/projected/cb069095-a5f5-45bd-9451-6f3b145a1219-kube-api-access-9mxxq\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:04.415022 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.414983    2587 generic.go:356] "Generic (PLEG): container finished" podID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerID="9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563" exitCode=0
Jul 13 19:12:04.415509 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.415038    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"
Jul 13 19:12:04.415509 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.415097    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" event={"ID":"cb069095-a5f5-45bd-9451-6f3b145a1219","Type":"ContainerDied","Data":"9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563"}
Jul 13 19:12:04.415509 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.415165    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r" event={"ID":"cb069095-a5f5-45bd-9451-6f3b145a1219","Type":"ContainerDied","Data":"16e891f287d77666d2b6fefdf573519c7240f9f322e4e61c3fd6431929c1b3f9"}
Jul 13 19:12:04.415509 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.415247    2587 scope.go:117] "RemoveContainer" containerID="58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d"
Jul 13 19:12:04.424383 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.424357    2587 scope.go:117] "RemoveContainer" containerID="9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563"
Jul 13 19:12:04.432103 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.432077    2587 scope.go:117] "RemoveContainer" containerID="58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d"
Jul 13 19:12:04.432431 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:04.432408    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d\": container with ID starting with 58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d not found: ID does not exist" containerID="58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d"
Jul 13 19:12:04.432504 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.432442    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d"} err="failed to get container status \"58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d\": rpc error: code = NotFound desc = could not find container \"58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d\": container with ID starting with 58cf3822bda03928e4e9a1f0f4f0f5695ac9e99e6be4a01b1e92aacc3ce44b0d not found: ID does not exist"
Jul 13 19:12:04.432504 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.432461    2587 scope.go:117] "RemoveContainer" containerID="9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563"
Jul 13 19:12:04.432749 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:04.432729    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563\": container with ID starting with 9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563 not found: ID does not exist" containerID="9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563"
Jul 13 19:12:04.432794 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.432757    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563"} err="failed to get container status \"9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563\": rpc error: code = NotFound desc = could not find container \"9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563\": container with ID starting with 9cecda278508a39e525eec1de575f0194e446fd340b32fbdec0eeed698e38563 not found: ID does not exist"
Jul 13 19:12:04.439584 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.439557    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"]
Jul 13 19:12:04.446087 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.446054    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a16a-predictor-59bd97769f-hwr4r"]
Jul 13 19:12:04.506881 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.506842    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2947986d-5361-4e23-908d-c270c4637661" path="/var/lib/kubelet/pods/2947986d-5361-4e23-908d-c270c4637661/volumes"
Jul 13 19:12:04.507380 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:04.507359    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" path="/var/lib/kubelet/pods/cb069095-a5f5-45bd-9451-6f3b145a1219/volumes"
Jul 13 19:12:07.404824 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:07.404789    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:07.405642 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:07.405609    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 19:12:07.407737 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:07.407714    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:12:07.408242 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:07.408191    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 19:12:11.246810 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:11.246771    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:12:12.252846 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:12.252806    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:12:17.406565 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:17.406517    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 19:12:17.408863 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:17.408823    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 19:12:21.246950 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:21.246919    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:12:22.253112 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:22.253079    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:12:27.406519 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:27.406476    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 19:12:27.408808 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:27.408779    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 19:12:29.324747 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:29.324707    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:33.702436 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:33.702343    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:33.891241 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:33.891189    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:37.405761 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:37.405714    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 19:12:37.409130 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:37.409082    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 19:12:43.780884 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.780842    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:43.856154 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.856116    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"]
Jul 13 19:12:43.856425 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.856397    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container" containerID="cri-o://9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1" gracePeriod=30
Jul 13 19:12:43.856601 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.856538    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kube-rbac-proxy" containerID="cri-o://1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd" gracePeriod=30
Jul 13 19:12:43.888450 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.888410    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:43.896589 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.896562    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"]
Jul 13 19:12:43.896824 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.896802    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container" containerID="cri-o://9500e6d35342f77b21a10426f0e03ee27eb23966d34a3af5bbf91ddd62d9fec1" gracePeriod=30
Jul 13 19:12:43.896910 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.896856    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kube-rbac-proxy" containerID="cri-o://aec3b254fb88818db1906ffb54bc539a54145b42550e785b1027b45e88774561" gracePeriod=30
Jul 13 19:12:43.935888 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.935854    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"]
Jul 13 19:12:43.936279 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936260    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kube-rbac-proxy"
Jul 13 19:12:43.936370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936280    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kube-rbac-proxy"
Jul 13 19:12:43.936370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936312    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container"
Jul 13 19:12:43.936370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936321    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container"
Jul 13 19:12:43.936370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936334    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kube-rbac-proxy"
Jul 13 19:12:43.936370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936342    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kube-rbac-proxy"
Jul 13 19:12:43.936370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936356    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container"
Jul 13 19:12:43.936370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936364    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container"
Jul 13 19:12:43.936704 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936451    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kube-rbac-proxy"
Jul 13 19:12:43.936704 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936467    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb069095-a5f5-45bd-9451-6f3b145a1219" containerName="kserve-container"
Jul 13 19:12:43.936704 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936478    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kserve-container"
Jul 13 19:12:43.936704 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.936488    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2947986d-5361-4e23-908d-c270c4637661" containerName="kube-rbac-proxy"
Jul 13 19:12:43.940979 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.940956    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:43.943464 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.943438    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ef1db-kube-rbac-proxy-sar-config\""
Jul 13 19:12:43.943555 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.943461    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ef1db-predictor-serving-cert\""
Jul 13 19:12:43.949731 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.949709    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"]
Jul 13 19:12:43.953579 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.953554    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf44r\" (UniqueName: \"kubernetes.io/projected/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-api-access-kf44r\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:43.953691 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.953638    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:43.953753 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.953716    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:43.985493 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:43.985458    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:44.026601 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.026564    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"]
Jul 13 19:12:44.029974 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.029954    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.032774 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.032720    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ef1db-predictor-serving-cert\""
Jul 13 19:12:44.032774 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.032751    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ef1db-kube-rbac-proxy-sar-config\""
Jul 13 19:12:44.038221 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.038186    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"]
Jul 13 19:12:44.054126 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.054095    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.054283 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.054148    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:44.054283 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.054221    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzss\" (UniqueName: \"kubernetes.io/projected/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-api-access-ghzss\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.054283 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:44.054264    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-serving-cert: secret "success-200-isvc-ef1db-predictor-serving-cert" not found
Jul 13 19:12:44.054283 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.054279    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf44r\" (UniqueName: \"kubernetes.io/projected/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-api-access-kf44r\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:44.054422 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:44.054312    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls podName:9217e6f3-d8a9-4158-83ce-68156f4f0caa nodeName:}" failed. No retries permitted until 2026-07-13 19:12:44.554297111 +0000 UTC m=+1310.794597189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls") pod "success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" (UID: "9217e6f3-d8a9-4158-83ce-68156f4f0caa") : secret "success-200-isvc-ef1db-predictor-serving-cert" not found
Jul 13 19:12:44.054422 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.054336    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.054422 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.054374    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:44.054926 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.054909    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:44.056083 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.056062    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:44.063768 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.063743    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf44r\" (UniqueName: \"kubernetes.io/projected/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-api-access-kf44r\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:44.124901 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.124867    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:44.155142 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.155103    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.155348 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.155169    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzss\" (UniqueName: \"kubernetes.io/projected/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-api-access-ghzss\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.155348 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.155221    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.155348 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:44.155312    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-serving-cert: secret "error-404-isvc-ef1db-predictor-serving-cert" not found
Jul 13 19:12:44.155510 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:44.155391    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls podName:7e30082a-3681-4bbc-8ad0-6e5067149ce5 nodeName:}" failed. No retries permitted until 2026-07-13 19:12:44.655370844 +0000 UTC m=+1310.895670936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls") pod "error-404-isvc-ef1db-predictor-867dc684b7-94jfl" (UID: "7e30082a-3681-4bbc-8ad0-6e5067149ce5") : secret "error-404-isvc-ef1db-predictor-serving-cert" not found
Jul 13 19:12:44.155864 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.155846    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.169614 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.169588    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzss\" (UniqueName: \"kubernetes.io/projected/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-api-access-ghzss\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.226400 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.226360    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:44.557729 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.557684    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:44.560132 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.560104    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls\") pod \"success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") " pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:44.583931 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.583898    2587 generic.go:356] "Generic (PLEG): container finished" podID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerID="aec3b254fb88818db1906ffb54bc539a54145b42550e785b1027b45e88774561" exitCode=2
Jul 13 19:12:44.584107 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.583969    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" event={"ID":"7377b88d-c036-4ff7-89de-fab1bc293e1a","Type":"ContainerDied","Data":"aec3b254fb88818db1906ffb54bc539a54145b42550e785b1027b45e88774561"}
Jul 13 19:12:44.585548 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.585530    2587 generic.go:356] "Generic (PLEG): container finished" podID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerID="1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd" exitCode=2
Jul 13 19:12:44.585617 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.585600    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" event={"ID":"8692dd78-4ecc-4590-80b2-f69841a4e023","Type":"ContainerDied","Data":"1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd"}
Jul 13 19:12:44.658529 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.658480    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.660919 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.660897    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls\") pod \"error-404-isvc-ef1db-predictor-867dc684b7-94jfl\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") " pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.851825 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.851723    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:44.940980 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.940947    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:44.998390 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:44.998352    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"]
Jul 13 19:12:45.002338 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:12:45.002292    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9217e6f3_d8a9_4158_83ce_68156f4f0caa.slice/crio-8698a5e3c54d585258c217db2e96736d4ffa0f829363d40c0f57676b95cbe892 WatchSource:0}: Error finding container 8698a5e3c54d585258c217db2e96736d4ffa0f829363d40c0f57676b95cbe892: Status 404 returned error can't find the container with id 8698a5e3c54d585258c217db2e96736d4ffa0f829363d40c0f57676b95cbe892
Jul 13 19:12:45.092556 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.092503    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"]
Jul 13 19:12:45.099359 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:12:45.099329    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e30082a_3681_4bbc_8ad0_6e5067149ce5.slice/crio-2a1f5a5b4ea825c7c739543a4735187feae54550d596bacd8e58628a03fedcb4 WatchSource:0}: Error finding container 2a1f5a5b4ea825c7c739543a4735187feae54550d596bacd8e58628a03fedcb4: Status 404 returned error can't find the container with id 2a1f5a5b4ea825c7c739543a4735187feae54550d596bacd8e58628a03fedcb4
Jul 13 19:12:45.591662 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.591624    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" event={"ID":"9217e6f3-d8a9-4158-83ce-68156f4f0caa","Type":"ContainerStarted","Data":"9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b"}
Jul 13 19:12:45.591863 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.591693    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:45.591863 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.591724    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" event={"ID":"9217e6f3-d8a9-4158-83ce-68156f4f0caa","Type":"ContainerStarted","Data":"cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5"}
Jul 13 19:12:45.591863 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.591753    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" event={"ID":"9217e6f3-d8a9-4158-83ce-68156f4f0caa","Type":"ContainerStarted","Data":"8698a5e3c54d585258c217db2e96736d4ffa0f829363d40c0f57676b95cbe892"}
Jul 13 19:12:45.591863 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.591788    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:45.593306 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.593270    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:45.593602 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.593582    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" event={"ID":"7e30082a-3681-4bbc-8ad0-6e5067149ce5","Type":"ContainerStarted","Data":"55257e17baaaa30d133a85d2d0541e7b45b57a41e624b792596c83e7a2cc03d8"}
Jul 13 19:12:45.593665 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.593625    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" event={"ID":"7e30082a-3681-4bbc-8ad0-6e5067149ce5","Type":"ContainerStarted","Data":"91c72d5d227fb74e3301b7f7fb8660999c7067f1ab5291cc31cdbf8ba5aa55c1"}
Jul 13 19:12:45.593665 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.593649    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:45.593746 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.593669    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" event={"ID":"7e30082a-3681-4bbc-8ad0-6e5067149ce5","Type":"ContainerStarted","Data":"2a1f5a5b4ea825c7c739543a4735187feae54550d596bacd8e58628a03fedcb4"}
Jul 13 19:12:45.593746 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.593720    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:45.594860 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.594827    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:12:45.611352 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.611291    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podStartSLOduration=2.611272991 podStartE2EDuration="2.611272991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:12:45.61073615 +0000 UTC m=+1311.851036250" watchObservedRunningTime="2026-07-13 19:12:45.611272991 +0000 UTC m=+1311.851573095"
Jul 13 19:12:45.629793 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:45.629736    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podStartSLOduration=1.629720523 podStartE2EDuration="1.629720523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:12:45.6276523 +0000 UTC m=+1311.867952399" watchObservedRunningTime="2026-07-13 19:12:45.629720523 +0000 UTC m=+1311.870021107"
Jul 13 19:12:46.241465 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:46.241418    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 13 19:12:46.598569 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:46.598526    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:46.598741 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:46.598649    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:12:47.247539 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.247491    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 13 19:12:47.406174 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.405707    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 19:12:47.412152 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.412070    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 19:12:47.432258 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:47.432226    2587 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7377b88d_c036_4ff7_89de_fab1bc293e1a.slice/crio-9500e6d35342f77b21a10426f0e03ee27eb23966d34a3af5bbf91ddd62d9fec1.scope\": RecentStats: unable to find data in memory cache]"
Jul 13 19:12:47.611244 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.606353    2587 generic.go:356] "Generic (PLEG): container finished" podID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerID="9500e6d35342f77b21a10426f0e03ee27eb23966d34a3af5bbf91ddd62d9fec1" exitCode=0
Jul 13 19:12:47.611244 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.606386    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" event={"ID":"7377b88d-c036-4ff7-89de-fab1bc293e1a","Type":"ContainerDied","Data":"9500e6d35342f77b21a10426f0e03ee27eb23966d34a3af5bbf91ddd62d9fec1"}
Jul 13 19:12:47.643553 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.643527    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:12:47.684388 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.684356    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7377b88d-c036-4ff7-89de-fab1bc293e1a-proxy-tls\") pod \"7377b88d-c036-4ff7-89de-fab1bc293e1a\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") "
Jul 13 19:12:47.684543 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.684431    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-rbac-proxy-sar-config\") pod \"7377b88d-c036-4ff7-89de-fab1bc293e1a\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") "
Jul 13 19:12:47.684634 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.684613    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl5fm\" (UniqueName: \"kubernetes.io/projected/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-api-access-rl5fm\") pod \"7377b88d-c036-4ff7-89de-fab1bc293e1a\" (UID: \"7377b88d-c036-4ff7-89de-fab1bc293e1a\") "
Jul 13 19:12:47.684726 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.684709    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7377b88d-c036-4ff7-89de-fab1bc293e1a" (UID: "7377b88d-c036-4ff7-89de-fab1bc293e1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:12:47.684866 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.684855    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:47.686463 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.686442    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7377b88d-c036-4ff7-89de-fab1bc293e1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7377b88d-c036-4ff7-89de-fab1bc293e1a" (UID: "7377b88d-c036-4ff7-89de-fab1bc293e1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:12:47.686645 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.686626    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-api-access-rl5fm" (OuterVolumeSpecName: "kube-api-access-rl5fm") pod "7377b88d-c036-4ff7-89de-fab1bc293e1a" (UID: "7377b88d-c036-4ff7-89de-fab1bc293e1a"). InnerVolumeSpecName "kube-api-access-rl5fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:12:47.786243 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.786183    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5fm\" (UniqueName: \"kubernetes.io/projected/7377b88d-c036-4ff7-89de-fab1bc293e1a-kube-api-access-rl5fm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:47.786243 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:47.786244    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7377b88d-c036-4ff7-89de-fab1bc293e1a-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.097692 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.097668    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:12:48.188775 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.188739    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55mlc\" (UniqueName: \"kubernetes.io/projected/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-api-access-55mlc\") pod \"8692dd78-4ecc-4590-80b2-f69841a4e023\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") "
Jul 13 19:12:48.188775 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.188784    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls\") pod \"8692dd78-4ecc-4590-80b2-f69841a4e023\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") "
Jul 13 19:12:48.188989 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.188820    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-rbac-proxy-sar-config\") pod \"8692dd78-4ecc-4590-80b2-f69841a4e023\" (UID: \"8692dd78-4ecc-4590-80b2-f69841a4e023\") "
Jul 13 19:12:48.189256 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.189197    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8692dd78-4ecc-4590-80b2-f69841a4e023" (UID: "8692dd78-4ecc-4590-80b2-f69841a4e023"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:12:48.190878 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.190857    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8692dd78-4ecc-4590-80b2-f69841a4e023" (UID: "8692dd78-4ecc-4590-80b2-f69841a4e023"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:12:48.190970 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.190916    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-api-access-55mlc" (OuterVolumeSpecName: "kube-api-access-55mlc") pod "8692dd78-4ecc-4590-80b2-f69841a4e023" (UID: "8692dd78-4ecc-4590-80b2-f69841a4e023"). InnerVolumeSpecName "kube-api-access-55mlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:12:48.289541 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.289490    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8692dd78-4ecc-4590-80b2-f69841a4e023-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.289541 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.289533    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.289541 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.289544    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55mlc\" (UniqueName: \"kubernetes.io/projected/8692dd78-4ecc-4590-80b2-f69841a4e023-kube-api-access-55mlc\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.613418 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.613387    2587 generic.go:356] "Generic (PLEG): container finished" podID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerID="9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1" exitCode=0
Jul 13 19:12:48.613553 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.613430    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"
Jul 13 19:12:48.613553 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.613463    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" event={"ID":"8692dd78-4ecc-4590-80b2-f69841a4e023","Type":"ContainerDied","Data":"9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1"}
Jul 13 19:12:48.613553 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.613529    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d" event={"ID":"8692dd78-4ecc-4590-80b2-f69841a4e023","Type":"ContainerDied","Data":"d7d7c0a84a7a64ed9ff1e117af82bfd11358b7b9c25f9bae0af22430214fed07"}
Jul 13 19:12:48.613699 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.613569    2587 scope.go:117] "RemoveContainer" containerID="1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd"
Jul 13 19:12:48.615323 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.615294    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh" event={"ID":"7377b88d-c036-4ff7-89de-fab1bc293e1a","Type":"ContainerDied","Data":"fd89e31aebbb88712d7638f4ca76b8c7022350084aac0ae0cc7c6ade62266bff"}
Jul 13 19:12:48.615444 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.615325    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"
Jul 13 19:12:48.622419 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.622399    2587 scope.go:117] "RemoveContainer" containerID="9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1"
Jul 13 19:12:48.630197 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.630126    2587 scope.go:117] "RemoveContainer" containerID="1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd"
Jul 13 19:12:48.630485 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:48.630461    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd\": container with ID starting with 1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd not found: ID does not exist" containerID="1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd"
Jul 13 19:12:48.630571 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.630493    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd"} err="failed to get container status \"1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd\": rpc error: code = NotFound desc = could not find container \"1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd\": container with ID starting with 1977b584ec30f547a7812e953c565bb6ba498b9480d7a9393d3ac8bd89ba3fdd not found: ID does not exist"
Jul 13 19:12:48.630571 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.630514    2587 scope.go:117] "RemoveContainer" containerID="9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1"
Jul 13 19:12:48.630802 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:12:48.630761    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1\": container with ID starting with 9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1 not found: ID does not exist" containerID="9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1"
Jul 13 19:12:48.630859 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.630797    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1"} err="failed to get container status \"9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1\": rpc error: code = NotFound desc = could not find container \"9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1\": container with ID starting with 9978e1ebc291e778e954b395b8512646fd146d80900f6aa7ad6acef51a61d3b1 not found: ID does not exist"
Jul 13 19:12:48.630859 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.630815    2587 scope.go:117] "RemoveContainer" containerID="aec3b254fb88818db1906ffb54bc539a54145b42550e785b1027b45e88774561"
Jul 13 19:12:48.631855 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.631837    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"]
Jul 13 19:12:48.637993 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.637967    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68f71-predictor-dc77967f4-s6d9d"]
Jul 13 19:12:48.639022 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.638982    2587 scope.go:117] "RemoveContainer" containerID="9500e6d35342f77b21a10426f0e03ee27eb23966d34a3af5bbf91ddd62d9fec1"
Jul 13 19:12:48.650653 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.650631    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"]
Jul 13 19:12:48.652593 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:48.652573    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-68f71-predictor-6478fb4ccf-w9blh"]
Jul 13 19:12:50.505817 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:50.505783    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" path="/var/lib/kubelet/pods/7377b88d-c036-4ff7-89de-fab1bc293e1a/volumes"
Jul 13 19:12:50.506237 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:50.506220    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" path="/var/lib/kubelet/pods/8692dd78-4ecc-4590-80b2-f69841a4e023/volumes"
Jul 13 19:12:51.603334 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:51.603302    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:12:51.603874 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:51.603844    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:12:51.604135 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:51.604115    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:12:51.604603 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:51.604578    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:57.406367 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:57.406333    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:12:57.409337 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:12:57.409312    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:13:01.603952 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:01.603907    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:01.604575 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:01.604541    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:13:09.683036 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:09.682996    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:09.815564 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:09.815532    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:11.604599 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:11.604554    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:11.605018 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:11.604695    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:13:19.786089 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.785992    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:19.869602 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.869570    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"]
Jul 13 19:13:19.869823 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.869798    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container" containerID="cri-o://a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e" gracePeriod=30
Jul 13 19:13:19.869924 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.869863    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kube-rbac-proxy" containerID="cri-o://70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f" gracePeriod=30
Jul 13 19:13:19.899948 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.899873    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:19.930079 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.930045    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"]
Jul 13 19:13:19.930379 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.930352    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kube-rbac-proxy" containerID="cri-o://70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4" gracePeriod=30
Jul 13 19:13:19.930668 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.930329    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container" containerID="cri-o://2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e" gracePeriod=30
Jul 13 19:13:19.958180 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958142    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"]
Jul 13 19:13:19.958535 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958519    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kube-rbac-proxy"
Jul 13 19:13:19.958586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958539    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kube-rbac-proxy"
Jul 13 19:13:19.958586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958554    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container"
Jul 13 19:13:19.958586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958560    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container"
Jul 13 19:13:19.958586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958571    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container"
Jul 13 19:13:19.958586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958577    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container"
Jul 13 19:13:19.958586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958582    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kube-rbac-proxy"
Jul 13 19:13:19.958586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958587    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kube-rbac-proxy"
Jul 13 19:13:19.958793 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958646    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kube-rbac-proxy"
Jul 13 19:13:19.958793 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958655    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kube-rbac-proxy"
Jul 13 19:13:19.958793 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958664    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7377b88d-c036-4ff7-89de-fab1bc293e1a" containerName="kserve-container"
Jul 13 19:13:19.958793 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.958670    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8692dd78-4ecc-4590-80b2-f69841a4e023" containerName="kserve-container"
Jul 13 19:13:19.961899 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.961877    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:19.964729 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.964703    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a68dc-kube-rbac-proxy-sar-config\""
Jul 13 19:13:19.964729 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.964726    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a68dc-predictor-serving-cert\""
Jul 13 19:13:19.970049 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.970024    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"]
Jul 13 19:13:19.974390 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:19.974365    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:20.045975 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.045929    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:20.055447 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.055418    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"]
Jul 13 19:13:20.058877 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.058857    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.061641 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.061613    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a68dc-predictor-serving-cert\""
Jul 13 19:13:20.061641 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.061629    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a68dc-kube-rbac-proxy-sar-config\""
Jul 13 19:13:20.064535 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.064513    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.064648 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.064579    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ec8725-5321-47b2-ab02-605a24f0fdca-proxy-tls\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.064700 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.064641    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w94g\" (UniqueName: \"kubernetes.io/projected/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-api-access-9w94g\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.069229 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.069192    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"]
Jul 13 19:13:20.153524 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.153484    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:20.165640 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.165599    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w94g\" (UniqueName: \"kubernetes.io/projected/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-api-access-9w94g\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.165833 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.165655    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.165833 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.165694    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.165833 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.165721    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e26c55-973f-41d1-bfb5-c1af71938dff-proxy-tls\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.165833 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.165752    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnpq\" (UniqueName: \"kubernetes.io/projected/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-api-access-zpnpq\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.165833 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.165797    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ec8725-5321-47b2-ab02-605a24f0fdca-proxy-tls\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.166386 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.166366    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.168322 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.168298    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ec8725-5321-47b2-ab02-605a24f0fdca-proxy-tls\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.174031 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.173999    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w94g\" (UniqueName: \"kubernetes.io/projected/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-api-access-9w94g\") pod \"success-200-isvc-a68dc-predictor-654cb8897d-4l8c8\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") " pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.252408 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.252369    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:20.267022 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.266994    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.267138 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.267044    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e26c55-973f-41d1-bfb5-c1af71938dff-proxy-tls\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.267138 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.267080    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnpq\" (UniqueName: \"kubernetes.io/projected/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-api-access-zpnpq\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.267748 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.267726    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.269504 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.269479    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e26c55-973f-41d1-bfb5-c1af71938dff-proxy-tls\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.273606 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.273589    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.276335 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.276315    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnpq\" (UniqueName: \"kubernetes.io/projected/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-api-access-zpnpq\") pod \"error-404-isvc-a68dc-predictor-8656958995-dh6rm\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") " pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.369686 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.369651    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.412230 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.411245    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"]
Jul 13 19:13:20.414544 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:13:20.414506    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ec8725_5321_47b2_ab02_605a24f0fdca.slice/crio-fad3749a582ecc56fd89d2397f14229c7e10326ad39a381357bdf8cd6c5bf5ba WatchSource:0}: Error finding container fad3749a582ecc56fd89d2397f14229c7e10326ad39a381357bdf8cd6c5bf5ba: Status 404 returned error can't find the container with id fad3749a582ecc56fd89d2397f14229c7e10326ad39a381357bdf8cd6c5bf5ba
Jul 13 19:13:20.510428 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.510334    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"]
Jul 13 19:13:20.513504 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:13:20.513477    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e26c55_973f_41d1_bfb5_c1af71938dff.slice/crio-5ea2989d8ea1b20596c633efaefcdf3511498b37a361346150e5cd5c04ac46b1 WatchSource:0}: Error finding container 5ea2989d8ea1b20596c633efaefcdf3511498b37a361346150e5cd5c04ac46b1: Status 404 returned error can't find the container with id 5ea2989d8ea1b20596c633efaefcdf3511498b37a361346150e5cd5c04ac46b1
Jul 13 19:13:20.757161 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.757118    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" event={"ID":"25ec8725-5321-47b2-ab02-605a24f0fdca","Type":"ContainerStarted","Data":"d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5"}
Jul 13 19:13:20.757367 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.757197    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" event={"ID":"25ec8725-5321-47b2-ab02-605a24f0fdca","Type":"ContainerStarted","Data":"bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957"}
Jul 13 19:13:20.757367 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.757245    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" event={"ID":"25ec8725-5321-47b2-ab02-605a24f0fdca","Type":"ContainerStarted","Data":"fad3749a582ecc56fd89d2397f14229c7e10326ad39a381357bdf8cd6c5bf5ba"}
Jul 13 19:13:20.757367 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.757339    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:20.758939 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.758913    2587 generic.go:356] "Generic (PLEG): container finished" podID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerID="70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4" exitCode=2
Jul 13 19:13:20.759076 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.758990    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" event={"ID":"91e5a392-9660-43eb-8494-ee0c8eb2702e","Type":"ContainerDied","Data":"70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4"}
Jul 13 19:13:20.760600 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.760578    2587 generic.go:356] "Generic (PLEG): container finished" podID="686d93a4-2a33-4158-be50-5531c78df11f" containerID="70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f" exitCode=2
Jul 13 19:13:20.760717 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.760650    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" event={"ID":"686d93a4-2a33-4158-be50-5531c78df11f","Type":"ContainerDied","Data":"70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f"}
Jul 13 19:13:20.762374 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.762353    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" event={"ID":"a9e26c55-973f-41d1-bfb5-c1af71938dff","Type":"ContainerStarted","Data":"1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685"}
Jul 13 19:13:20.762469 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.762396    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" event={"ID":"a9e26c55-973f-41d1-bfb5-c1af71938dff","Type":"ContainerStarted","Data":"8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5"}
Jul 13 19:13:20.762469 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.762416    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" event={"ID":"a9e26c55-973f-41d1-bfb5-c1af71938dff","Type":"ContainerStarted","Data":"5ea2989d8ea1b20596c633efaefcdf3511498b37a361346150e5cd5c04ac46b1"}
Jul 13 19:13:20.762469 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.762448    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:20.777912 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.777864    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podStartSLOduration=1.777849299 podStartE2EDuration="1.777849299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:13:20.777268033 +0000 UTC m=+1347.017568134" watchObservedRunningTime="2026-07-13 19:13:20.777849299 +0000 UTC m=+1347.018149399"
Jul 13 19:13:20.794846 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:20.794791    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podStartSLOduration=0.794775456 podStartE2EDuration="794.775456ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:13:20.794046827 +0000 UTC m=+1347.034346980" watchObservedRunningTime="2026-07-13 19:13:20.794775456 +0000 UTC m=+1347.035075618"
Jul 13 19:13:21.603903 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:21.603858    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:21.604734 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:21.604693    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:13:21.766799 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:21.766762    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:21.766985 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:21.766828    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:21.767766 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:21.767738    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 19:13:21.767837 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:21.767738    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 19:13:22.400295 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:22.400245    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 13 19:13:22.400773 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:22.400247    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 13 19:13:22.770605 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:22.770555    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 19:13:22.770773 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:22.770617    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 19:13:23.630435 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.630405    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:13:23.775334 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.775305    2587 generic.go:356] "Generic (PLEG): container finished" podID="686d93a4-2a33-4158-be50-5531c78df11f" containerID="a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e" exitCode=0
Jul 13 19:13:23.775534 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.775355    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"
Jul 13 19:13:23.775534 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.775401    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" event={"ID":"686d93a4-2a33-4158-be50-5531c78df11f","Type":"ContainerDied","Data":"a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e"}
Jul 13 19:13:23.775534 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.775474    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx" event={"ID":"686d93a4-2a33-4158-be50-5531c78df11f","Type":"ContainerDied","Data":"a5b1e250cac0548908c3ce310ef4118cb3cb8c32f4d59bc6fb0576ea7d4d890a"}
Jul 13 19:13:23.775534 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.775515    2587 scope.go:117] "RemoveContainer" containerID="70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f"
Jul 13 19:13:23.783570 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.783550    2587 scope.go:117] "RemoveContainer" containerID="a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e"
Jul 13 19:13:23.790637 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.790620    2587 scope.go:117] "RemoveContainer" containerID="70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f"
Jul 13 19:13:23.790894 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:13:23.790874    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f\": container with ID starting with 70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f not found: ID does not exist" containerID="70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f"
Jul 13 19:13:23.790940 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.790904    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f"} err="failed to get container status \"70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f\": rpc error: code = NotFound desc = could not find container \"70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f\": container with ID starting with 70ac4bcee24800d06c0505ce02a54eeb8aa26affc7ff37cf9b6d7f7bfd353b1f not found: ID does not exist"
Jul 13 19:13:23.790940 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.790922    2587 scope.go:117] "RemoveContainer" containerID="a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e"
Jul 13 19:13:23.791158 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:13:23.791142    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e\": container with ID starting with a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e not found: ID does not exist" containerID="a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e"
Jul 13 19:13:23.791226 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.791170    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e"} err="failed to get container status \"a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e\": rpc error: code = NotFound desc = could not find container \"a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e\": container with ID starting with a8f55ef64e3258997185b79912996ae1e4b2be279efc4fa0c73ae98f7cf0864e not found: ID does not exist"
Jul 13 19:13:23.795432 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.795412    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/686d93a4-2a33-4158-be50-5531c78df11f-kube-rbac-proxy-sar-config\") pod \"686d93a4-2a33-4158-be50-5531c78df11f\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") "
Jul 13 19:13:23.795529 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.795498    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls\") pod \"686d93a4-2a33-4158-be50-5531c78df11f\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") "
Jul 13 19:13:23.795586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.795552    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r7lp\" (UniqueName: \"kubernetes.io/projected/686d93a4-2a33-4158-be50-5531c78df11f-kube-api-access-4r7lp\") pod \"686d93a4-2a33-4158-be50-5531c78df11f\" (UID: \"686d93a4-2a33-4158-be50-5531c78df11f\") "
Jul 13 19:13:23.795762 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.795738    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686d93a4-2a33-4158-be50-5531c78df11f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "686d93a4-2a33-4158-be50-5531c78df11f" (UID: "686d93a4-2a33-4158-be50-5531c78df11f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:13:23.797544 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.797522    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "686d93a4-2a33-4158-be50-5531c78df11f" (UID: "686d93a4-2a33-4158-be50-5531c78df11f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:13:23.797616 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.797558    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686d93a4-2a33-4158-be50-5531c78df11f-kube-api-access-4r7lp" (OuterVolumeSpecName: "kube-api-access-4r7lp") pod "686d93a4-2a33-4158-be50-5531c78df11f" (UID: "686d93a4-2a33-4158-be50-5531c78df11f"). InnerVolumeSpecName "kube-api-access-4r7lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:13:23.896942 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.896908    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/686d93a4-2a33-4158-be50-5531c78df11f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:13:23.896942 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.896939    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/686d93a4-2a33-4158-be50-5531c78df11f-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:13:23.896942 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:23.896949    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r7lp\" (UniqueName: \"kubernetes.io/projected/686d93a4-2a33-4158-be50-5531c78df11f-kube-api-access-4r7lp\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:13:24.098525 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:24.098488    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"]
Jul 13 19:13:24.106560 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:24.106523    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e56dd-predictor-6b959756d6-cslzx"]
Jul 13 19:13:24.507240 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:24.507126    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="686d93a4-2a33-4158-be50-5531c78df11f" path="/var/lib/kubelet/pods/686d93a4-2a33-4158-be50-5531c78df11f/volumes"
Jul 13 19:13:27.087891 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.087861    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:13:27.226836 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.226744    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e5a392-9660-43eb-8494-ee0c8eb2702e-proxy-tls\") pod \"91e5a392-9660-43eb-8494-ee0c8eb2702e\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") "
Jul 13 19:13:27.226836 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.226789    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mwt9\" (UniqueName: \"kubernetes.io/projected/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-api-access-6mwt9\") pod \"91e5a392-9660-43eb-8494-ee0c8eb2702e\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") "
Jul 13 19:13:27.226836 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.226826    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-rbac-proxy-sar-config\") pod \"91e5a392-9660-43eb-8494-ee0c8eb2702e\" (UID: \"91e5a392-9660-43eb-8494-ee0c8eb2702e\") "
Jul 13 19:13:27.227225 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.227167    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91e5a392-9660-43eb-8494-ee0c8eb2702e" (UID: "91e5a392-9660-43eb-8494-ee0c8eb2702e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:13:27.229007 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.228979    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e5a392-9660-43eb-8494-ee0c8eb2702e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91e5a392-9660-43eb-8494-ee0c8eb2702e" (UID: "91e5a392-9660-43eb-8494-ee0c8eb2702e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:13:27.229119 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.228985    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-api-access-6mwt9" (OuterVolumeSpecName: "kube-api-access-6mwt9") pod "91e5a392-9660-43eb-8494-ee0c8eb2702e" (UID: "91e5a392-9660-43eb-8494-ee0c8eb2702e"). InnerVolumeSpecName "kube-api-access-6mwt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:13:27.327989 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.327945    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e5a392-9660-43eb-8494-ee0c8eb2702e-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:13:27.327989 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.327982    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mwt9\" (UniqueName: \"kubernetes.io/projected/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-api-access-6mwt9\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:13:27.327989 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.327996    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e5a392-9660-43eb-8494-ee0c8eb2702e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:13:27.774465 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.774423    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:13:27.774833 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.774810    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:13:27.774913 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.774885    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 19:13:27.775323 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.775298    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 19:13:27.794669 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.794638    2587 generic.go:356] "Generic (PLEG): container finished" podID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerID="2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e" exitCode=0
Jul 13 19:13:27.794832 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.794689    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"
Jul 13 19:13:27.794832 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.794704    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" event={"ID":"91e5a392-9660-43eb-8494-ee0c8eb2702e","Type":"ContainerDied","Data":"2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e"}
Jul 13 19:13:27.794832 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.794748    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8" event={"ID":"91e5a392-9660-43eb-8494-ee0c8eb2702e","Type":"ContainerDied","Data":"fbed37baea00dc4df91e39c55dec0cea3d0659cfa1533b0fa6bb2d013d9320cd"}
Jul 13 19:13:27.794832 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.794775    2587 scope.go:117] "RemoveContainer" containerID="70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4"
Jul 13 19:13:27.804087 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.803984    2587 scope.go:117] "RemoveContainer" containerID="2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e"
Jul 13 19:13:27.813180 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.813155    2587 scope.go:117] "RemoveContainer" containerID="70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4"
Jul 13 19:13:27.813490 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:13:27.813467    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4\": container with ID starting with 70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4 not found: ID does not exist" containerID="70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4"
Jul 13 19:13:27.813539 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.813500    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4"} err="failed to get container status \"70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4\": rpc error: code = NotFound desc = could not find container \"70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4\": container with ID starting with 70f378bd888557bd9cbb7c18b4c5c05712028b112af1c2fbe9f739e16906cde4 not found: ID does not exist"
Jul 13 19:13:27.813539 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.813520    2587 scope.go:117] "RemoveContainer" containerID="2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e"
Jul 13 19:13:27.813750 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:13:27.813730    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e\": container with ID starting with 2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e not found: ID does not exist" containerID="2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e"
Jul 13 19:13:27.813795 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.813757    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e"} err="failed to get container status \"2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e\": rpc error: code = NotFound desc = could not find container \"2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e\": container with ID starting with 2b7d00c032cfcd436f9380e2906fa67a98c2d00b004fa71e76e3b088dafa3c3e not found: ID does not exist"
Jul 13 19:13:27.824497 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.824472    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"]
Jul 13 19:13:27.828154 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:27.828131    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e56dd-predictor-7f8878c579-l8lf8"]
Jul 13 19:13:28.507730 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:28.507680    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" path="/var/lib/kubelet/pods/91e5a392-9660-43eb-8494-ee0c8eb2702e/volumes"
Jul 13 19:13:31.604036 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:31.603985    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:31.604609 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:31.604575    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:13:37.775659 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:37.775613    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 19:13:37.776154 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:37.775613    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 19:13:41.604586 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:41.604546    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:13:41.605559 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:41.605540    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:13:47.778727 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:47.778679    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 19:13:47.779147 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:47.778741    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 19:13:54.031144 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:54.031089    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:54.130282 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:54.130194    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:57.774932 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:57.774885    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 19:13:57.775358 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:57.775235    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 19:13:59.656727 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:13:59.656693    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:07.775576 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:14:07.775535    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 19:14:07.776127 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:14:07.776108    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:14:17.776091 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:14:17.776058    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:14:30.011946 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:14:30.011909    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:30.200817 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:14:30.200771    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:33.578983 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:14:33.578947    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:54.401588 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:15:54.401560    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:15:54.404888 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:15:54.404862    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:18:59.194418 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:18:59.194379    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:20:54.426317 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:20:54.426290    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:20:54.429403 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:20:54.429380    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:22:08.766242 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.766184    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:08.850850 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.850814    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"]
Jul 13 19:22:08.851177 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.851117    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" containerID="cri-o://cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5" gracePeriod=30
Jul 13 19:22:08.851297 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.851142    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kube-rbac-proxy" containerID="cri-o://9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b" gracePeriod=30
Jul 13 19:22:08.871835 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.871788    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:08.910869 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.910830    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"]
Jul 13 19:22:08.911192 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911179    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container"
Jul 13 19:22:08.911272 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911194    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container"
Jul 13 19:22:08.911272 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911221    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container"
Jul 13 19:22:08.911272 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911226    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container"
Jul 13 19:22:08.911272 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911237    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kube-rbac-proxy"
Jul 13 19:22:08.911272 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911243    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kube-rbac-proxy"
Jul 13 19:22:08.911272 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911256    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kube-rbac-proxy"
Jul 13 19:22:08.911272 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911261    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kube-rbac-proxy"
Jul 13 19:22:08.911518 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911309    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kube-rbac-proxy"
Jul 13 19:22:08.911518 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911319    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kube-rbac-proxy"
Jul 13 19:22:08.911518 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911326    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e5a392-9660-43eb-8494-ee0c8eb2702e" containerName="kserve-container"
Jul 13 19:22:08.911518 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.911332    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="686d93a4-2a33-4158-be50-5531c78df11f" containerName="kserve-container"
Jul 13 19:22:08.914406 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.914384    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:08.917115 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.917086    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b5afb-predictor-serving-cert\""
Jul 13 19:22:08.917257 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.917100    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b5afb-kube-rbac-proxy-sar-config\""
Jul 13 19:22:08.925898 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.925871    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"]
Jul 13 19:22:08.956961 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:08.956928    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:09.000463 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.000433    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"]
Jul 13 19:22:09.004126 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.004105    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.008671 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.008638    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b5afb-kube-rbac-proxy-sar-config\""
Jul 13 19:22:09.009065 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.008678    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b5afb-predictor-serving-cert\""
Jul 13 19:22:09.012118 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.012021    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"]
Jul 13 19:22:09.030540 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.030511    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwvz\" (UniqueName: \"kubernetes.io/projected/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-api-access-vtwvz\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.030713 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.030565    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.030713 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.030671    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.053838 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.053801    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"]
Jul 13 19:22:09.054065 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.054025    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" containerID="cri-o://91c72d5d227fb74e3301b7f7fb8660999c7067f1ab5291cc31cdbf8ba5aa55c1" gracePeriod=30
Jul 13 19:22:09.054154 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.054055    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kube-rbac-proxy" containerID="cri-o://55257e17baaaa30d133a85d2d0541e7b45b57a41e624b792596c83e7a2cc03d8" gracePeriod=30
Jul 13 19:22:09.120435 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.120403    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:09.131999 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.131967    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.132159 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.132017    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.132159 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.132097    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.132159 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:09.132131    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-serving-cert: secret "success-200-isvc-b5afb-predictor-serving-cert" not found
Jul 13 19:22:09.132369 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:09.132222    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls podName:893f2442-98f0-45b6-bccd-17a4f7826dbb nodeName:}" failed. No retries permitted until 2026-07-13 19:22:09.6321837 +0000 UTC m=+1875.872483778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls") pod "success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" (UID: "893f2442-98f0-45b6-bccd-17a4f7826dbb") : secret "success-200-isvc-b5afb-predictor-serving-cert" not found
Jul 13 19:22:09.132369 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.132131    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.132369 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.132258    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkjr\" (UniqueName: \"kubernetes.io/projected/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-api-access-5lkjr\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.132369 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.132324    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwvz\" (UniqueName: \"kubernetes.io/projected/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-api-access-vtwvz\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.132833 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.132809    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.143054 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.143028    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwvz\" (UniqueName: \"kubernetes.io/projected/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-api-access-vtwvz\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.185013 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.184976    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:09.233567 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.233527    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.233756 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.233596    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.233756 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.233619    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkjr\" (UniqueName: \"kubernetes.io/projected/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-api-access-5lkjr\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.233756 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:09.233686    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-serving-cert: secret "error-404-isvc-b5afb-predictor-serving-cert" not found
Jul 13 19:22:09.233917 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:09.233770    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls podName:6194efd1-4acd-483c-9c77-2dc1b5630ebe nodeName:}" failed. No retries permitted until 2026-07-13 19:22:09.733746854 +0000 UTC m=+1875.974046932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls") pod "error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" (UID: "6194efd1-4acd-483c-9c77-2dc1b5630ebe") : secret "error-404-isvc-b5afb-predictor-serving-cert" not found
Jul 13 19:22:09.234279 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.234260    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.249023 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.248990    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkjr\" (UniqueName: \"kubernetes.io/projected/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-api-access-5lkjr\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.295114 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.295030    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:09.389535 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.389482    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:09.637246 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.637109    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.639637 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.639610    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls\") pod \"success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") " pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.738094 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.738048    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.740587 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.740554    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls\") pod \"error-404-isvc-b5afb-predictor-6754b9b7-xxv6p\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") " pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.825123 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.825082    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:09.827927 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.827895    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerID="55257e17baaaa30d133a85d2d0541e7b45b57a41e624b792596c83e7a2cc03d8" exitCode=2
Jul 13 19:22:09.828072 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.827991    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" event={"ID":"7e30082a-3681-4bbc-8ad0-6e5067149ce5","Type":"ContainerDied","Data":"55257e17baaaa30d133a85d2d0541e7b45b57a41e624b792596c83e7a2cc03d8"}
Jul 13 19:22:09.829941 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.829910    2587 generic.go:356] "Generic (PLEG): container finished" podID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerID="9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b" exitCode=2
Jul 13 19:22:09.830033 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.829978    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" event={"ID":"9217e6f3-d8a9-4158-83ce-68156f4f0caa","Type":"ContainerDied","Data":"9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b"}
Jul 13 19:22:09.916617 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.916564    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:09.966140 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:09.966108    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"]
Jul 13 19:22:09.968795 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:22:09.968747    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893f2442_98f0_45b6_bccd_17a4f7826dbb.slice/crio-5b0a12847e18c67595d20d8d93171c956c13fbd8f356655028ff6ba034125c41 WatchSource:0}: Error finding container 5b0a12847e18c67595d20d8d93171c956c13fbd8f356655028ff6ba034125c41: Status 404 returned error can't find the container with id 5b0a12847e18c67595d20d8d93171c956c13fbd8f356655028ff6ba034125c41
Jul 13 19:22:10.065073 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.065051    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"]
Jul 13 19:22:10.066980 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:22:10.066957    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6194efd1_4acd_483c_9c77_2dc1b5630ebe.slice/crio-0da7ca44cc85514738a4ab0ba31d3aa573b53c163c565fb046a4ae9242a1c5e7 WatchSource:0}: Error finding container 0da7ca44cc85514738a4ab0ba31d3aa573b53c163c565fb046a4ae9242a1c5e7: Status 404 returned error can't find the container with id 0da7ca44cc85514738a4ab0ba31d3aa573b53c163c565fb046a4ae9242a1c5e7
Jul 13 19:22:10.836176 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.836138    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" event={"ID":"893f2442-98f0-45b6-bccd-17a4f7826dbb","Type":"ContainerStarted","Data":"a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe"}
Jul 13 19:22:10.836649 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.836226    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" event={"ID":"893f2442-98f0-45b6-bccd-17a4f7826dbb","Type":"ContainerStarted","Data":"5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf"}
Jul 13 19:22:10.836649 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.836267    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" event={"ID":"893f2442-98f0-45b6-bccd-17a4f7826dbb","Type":"ContainerStarted","Data":"5b0a12847e18c67595d20d8d93171c956c13fbd8f356655028ff6ba034125c41"}
Jul 13 19:22:10.836649 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.836399    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:10.838078 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.838052    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" event={"ID":"6194efd1-4acd-483c-9c77-2dc1b5630ebe","Type":"ContainerStarted","Data":"e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f"}
Jul 13 19:22:10.838221 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.838099    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:10.838221 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.838119    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" event={"ID":"6194efd1-4acd-483c-9c77-2dc1b5630ebe","Type":"ContainerStarted","Data":"e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28"}
Jul 13 19:22:10.838221 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.838140    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" event={"ID":"6194efd1-4acd-483c-9c77-2dc1b5630ebe","Type":"ContainerStarted","Data":"0da7ca44cc85514738a4ab0ba31d3aa573b53c163c565fb046a4ae9242a1c5e7"}
Jul 13 19:22:10.838221 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.838161    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:10.839622 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.839601    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 19:22:10.856149 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.856101    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podStartSLOduration=2.856088228 podStartE2EDuration="2.856088228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:22:10.854756546 +0000 UTC m=+1877.095056653" watchObservedRunningTime="2026-07-13 19:22:10.856088228 +0000 UTC m=+1877.096388327"
Jul 13 19:22:10.874302 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:10.874251    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podStartSLOduration=2.874235594 podStartE2EDuration="2.874235594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:22:10.87364476 +0000 UTC m=+1877.113944862" watchObservedRunningTime="2026-07-13 19:22:10.874235594 +0000 UTC m=+1877.114535694"
Jul 13 19:22:11.599402 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:11.599359    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 13 19:22:11.599611 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:11.599359    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 13 19:22:11.604905 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:11.604867    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:22:11.605025 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:11.604872    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:22:11.841889 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:11.841845    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 19:22:11.842383 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:11.841989    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:11.843354 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:11.843327    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 19:22:12.316251 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.316224    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:22:12.462485 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.462392    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls\") pod \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") "
Jul 13 19:22:12.462485 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.462473    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf44r\" (UniqueName: \"kubernetes.io/projected/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-api-access-kf44r\") pod \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") "
Jul 13 19:22:12.462721 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.462491    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-rbac-proxy-sar-config\") pod \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\" (UID: \"9217e6f3-d8a9-4158-83ce-68156f4f0caa\") "
Jul 13 19:22:12.462909 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.462885    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9217e6f3-d8a9-4158-83ce-68156f4f0caa" (UID: "9217e6f3-d8a9-4158-83ce-68156f4f0caa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:22:12.464458 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.464435    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9217e6f3-d8a9-4158-83ce-68156f4f0caa" (UID: "9217e6f3-d8a9-4158-83ce-68156f4f0caa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:22:12.464595 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.464575    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-api-access-kf44r" (OuterVolumeSpecName: "kube-api-access-kf44r") pod "9217e6f3-d8a9-4158-83ce-68156f4f0caa" (UID: "9217e6f3-d8a9-4158-83ce-68156f4f0caa"). InnerVolumeSpecName "kube-api-access-kf44r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:12.563842 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.563803    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf44r\" (UniqueName: \"kubernetes.io/projected/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-api-access-kf44r\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:12.563842 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.563835    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9217e6f3-d8a9-4158-83ce-68156f4f0caa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:12.563842 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.563845    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9217e6f3-d8a9-4158-83ce-68156f4f0caa-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:12.848401 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.848367    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerID="91c72d5d227fb74e3301b7f7fb8660999c7067f1ab5291cc31cdbf8ba5aa55c1" exitCode=0
Jul 13 19:22:12.848862 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.848472    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" event={"ID":"7e30082a-3681-4bbc-8ad0-6e5067149ce5","Type":"ContainerDied","Data":"91c72d5d227fb74e3301b7f7fb8660999c7067f1ab5291cc31cdbf8ba5aa55c1"}
Jul 13 19:22:12.850376 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.850346    2587 generic.go:356] "Generic (PLEG): container finished" podID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerID="cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5" exitCode=0
Jul 13 19:22:12.850489 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.850414    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"
Jul 13 19:22:12.850489 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.850428    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" event={"ID":"9217e6f3-d8a9-4158-83ce-68156f4f0caa","Type":"ContainerDied","Data":"cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5"}
Jul 13 19:22:12.850489 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.850486    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b" event={"ID":"9217e6f3-d8a9-4158-83ce-68156f4f0caa","Type":"ContainerDied","Data":"8698a5e3c54d585258c217db2e96736d4ffa0f829363d40c0f57676b95cbe892"}
Jul 13 19:22:12.850662 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.850510    2587 scope.go:117] "RemoveContainer" containerID="9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b"
Jul 13 19:22:12.851274 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.850987    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 19:22:12.860381 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.860358    2587 scope.go:117] "RemoveContainer" containerID="cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5"
Jul 13 19:22:12.869127 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.869106    2587 scope.go:117] "RemoveContainer" containerID="9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b"
Jul 13 19:22:12.869222 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.869121    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"]
Jul 13 19:22:12.869437 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:12.869420    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b\": container with ID starting with 9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b not found: ID does not exist" containerID="9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b"
Jul 13 19:22:12.869491 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.869444    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b"} err="failed to get container status \"9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b\": rpc error: code = NotFound desc = could not find container \"9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b\": container with ID starting with 9e6a1f6f6329ff76b5471cfb6ca9a692e3a3e1c8c80121c44309e8cf2dd8b60b not found: ID does not exist"
Jul 13 19:22:12.869491 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.869462    2587 scope.go:117] "RemoveContainer" containerID="cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5"
Jul 13 19:22:12.869721 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:12.869702    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5\": container with ID starting with cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5 not found: ID does not exist" containerID="cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5"
Jul 13 19:22:12.869765 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.869727    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5"} err="failed to get container status \"cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5\": rpc error: code = NotFound desc = could not find container \"cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5\": container with ID starting with cd0b7eb6e7c294fc85670cfd759ed6ba8033b418087e6a60761519e40b902ad5 not found: ID does not exist"
Jul 13 19:22:12.872554 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.872503    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef1db-predictor-5bdcf6bfc4-v2s9b"]
Jul 13 19:22:12.897672 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:12.897648    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:22:13.069244 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.069105    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls\") pod \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") "
Jul 13 19:22:13.069244 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.069192    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghzss\" (UniqueName: \"kubernetes.io/projected/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-api-access-ghzss\") pod \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") "
Jul 13 19:22:13.069483 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.069275    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-rbac-proxy-sar-config\") pod \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\" (UID: \"7e30082a-3681-4bbc-8ad0-6e5067149ce5\") "
Jul 13 19:22:13.069721 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.069697    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e30082a-3681-4bbc-8ad0-6e5067149ce5" (UID: "7e30082a-3681-4bbc-8ad0-6e5067149ce5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:22:13.071388 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.071361    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-api-access-ghzss" (OuterVolumeSpecName: "kube-api-access-ghzss") pod "7e30082a-3681-4bbc-8ad0-6e5067149ce5" (UID: "7e30082a-3681-4bbc-8ad0-6e5067149ce5"). InnerVolumeSpecName "kube-api-access-ghzss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:13.071388 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.071371    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e30082a-3681-4bbc-8ad0-6e5067149ce5" (UID: "7e30082a-3681-4bbc-8ad0-6e5067149ce5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:22:13.170087 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.170045    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.170087 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.170082    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e30082a-3681-4bbc-8ad0-6e5067149ce5-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.170087 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.170095    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzss\" (UniqueName: \"kubernetes.io/projected/7e30082a-3681-4bbc-8ad0-6e5067149ce5-kube-api-access-ghzss\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.669774 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.669724    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:13.855625 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.855588    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"
Jul 13 19:22:13.856091 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.855587    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl" event={"ID":"7e30082a-3681-4bbc-8ad0-6e5067149ce5","Type":"ContainerDied","Data":"2a1f5a5b4ea825c7c739543a4735187feae54550d596bacd8e58628a03fedcb4"}
Jul 13 19:22:13.856091 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.855731    2587 scope.go:117] "RemoveContainer" containerID="55257e17baaaa30d133a85d2d0541e7b45b57a41e624b792596c83e7a2cc03d8"
Jul 13 19:22:13.864171 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.864145    2587 scope.go:117] "RemoveContainer" containerID="91c72d5d227fb74e3301b7f7fb8660999c7067f1ab5291cc31cdbf8ba5aa55c1"
Jul 13 19:22:13.877513 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.877486    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"]
Jul 13 19:22:13.881479 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:13.881455    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ef1db-predictor-867dc684b7-94jfl"]
Jul 13 19:22:14.505077 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:14.505036    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" path="/var/lib/kubelet/pods/7e30082a-3681-4bbc-8ad0-6e5067149ce5/volumes"
Jul 13 19:22:14.505616 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:14.505594    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" path="/var/lib/kubelet/pods/9217e6f3-d8a9-4158-83ce-68156f4f0caa/volumes"
Jul 13 19:22:16.846074 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:16.846004    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:22:16.846498 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:16.846473    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 19:22:17.856046 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:17.856008    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:22:17.856705 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:17.856679    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 19:22:26.846726 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:26.846683    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 19:22:27.856809 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:27.856770    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 19:22:28.634182 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:28.634147    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:36.846580 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:36.846538    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 19:22:37.857017 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:37.856980    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 19:22:44.596379 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.596329    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:44.674849 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.674816    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"]
Jul 13 19:22:44.675083 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.675057    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" containerID="cri-o://bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957" gracePeriod=30
Jul 13 19:22:44.675217 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.675175    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kube-rbac-proxy" containerID="cri-o://d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5" gracePeriod=30
Jul 13 19:22:44.710859 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.710822    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:44.726676 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.726646    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"]
Jul 13 19:22:44.726912 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.726887    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container" containerID="cri-o://8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5" gracePeriod=30
Jul 13 19:22:44.726987 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.726928    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kube-rbac-proxy" containerID="cri-o://1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685" gracePeriod=30
Jul 13 19:22:44.739738 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.739710    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"]
Jul 13 19:22:44.740078 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740062    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container"
Jul 13 19:22:44.740164 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740082    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container"
Jul 13 19:22:44.740164 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740093    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kube-rbac-proxy"
Jul 13 19:22:44.740164 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740102    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kube-rbac-proxy"
Jul 13 19:22:44.740164 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740138    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kube-rbac-proxy"
Jul 13 19:22:44.740164 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740147    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kube-rbac-proxy"
Jul 13 19:22:44.740432 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740166    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container"
Jul 13 19:22:44.740432 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740174    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container"
Jul 13 19:22:44.740432 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740292    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kube-rbac-proxy"
Jul 13 19:22:44.740432 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740306    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9217e6f3-d8a9-4158-83ce-68156f4f0caa" containerName="kserve-container"
Jul 13 19:22:44.740432 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740317    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kserve-container"
Jul 13 19:22:44.740432 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.740327    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e30082a-3681-4bbc-8ad0-6e5067149ce5" containerName="kube-rbac-proxy"
Jul 13 19:22:44.743347 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.743328    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.747916 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.747714    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4dc86-predictor-serving-cert\""
Jul 13 19:22:44.748028 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.747952    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4dc86-kube-rbac-proxy-sar-config\""
Jul 13 19:22:44.755412 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.755391    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"]
Jul 13 19:22:44.777168 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.777143    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:44.829649 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.829614    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.829649 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.829661    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.829931 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.829741    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l6bn\" (UniqueName: \"kubernetes.io/projected/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-api-access-4l6bn\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.840987 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.840960    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:44.848214 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.848116    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"]
Jul 13 19:22:44.851714 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.851685    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:44.854267 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.854234    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4dc86-predictor-serving-cert\""
Jul 13 19:22:44.854558 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.854537    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4dc86-kube-rbac-proxy-sar-config\""
Jul 13 19:22:44.861391 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.861361    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"]
Jul 13 19:22:44.930398 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.930359    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l6bn\" (UniqueName: \"kubernetes.io/projected/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-api-access-4l6bn\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.930602 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.930411    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:44.930602 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.930437    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:44.930602 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.930482    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.930602 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.930532    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l65w\" (UniqueName: \"kubernetes.io/projected/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-api-access-5l65w\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:44.930602 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.930590    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.930899 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:44.930666    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-serving-cert: secret "success-200-isvc-4dc86-predictor-serving-cert" not found
Jul 13 19:22:44.930899 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:44.930722    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls podName:c0b288cb-227a-4c81-940b-a72e93e2405b nodeName:}" failed. No retries permitted until 2026-07-13 19:22:45.430707351 +0000 UTC m=+1911.671007429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls") pod "success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" (UID: "c0b288cb-227a-4c81-940b-a72e93e2405b") : secret "success-200-isvc-4dc86-predictor-serving-cert" not found
Jul 13 19:22:44.931236 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.931188    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.943099 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.943070    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l6bn\" (UniqueName: \"kubernetes.io/projected/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-api-access-4l6bn\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:44.981015 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.980979    2587 generic.go:356] "Generic (PLEG): container finished" podID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerID="d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5" exitCode=2
Jul 13 19:22:44.981225 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.981056    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" event={"ID":"25ec8725-5321-47b2-ab02-605a24f0fdca","Type":"ContainerDied","Data":"d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5"}
Jul 13 19:22:44.982753 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.982730    2587 generic.go:356] "Generic (PLEG): container finished" podID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerID="1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685" exitCode=2
Jul 13 19:22:44.982878 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:44.982818    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" event={"ID":"a9e26c55-973f-41d1-bfb5-c1af71938dff","Type":"ContainerDied","Data":"1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685"}
Jul 13 19:22:45.027762 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.027711    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:45.031762 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.031732    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l65w\" (UniqueName: \"kubernetes.io/projected/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-api-access-5l65w\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:45.031924 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.031852    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:45.031924 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.031881    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:45.032040 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:45.031997    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-serving-cert: secret "error-404-isvc-4dc86-predictor-serving-cert" not found
Jul 13 19:22:45.032090 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:45.032056    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls podName:5cf5f081-81e8-4cd2-a14e-4d235593ae37 nodeName:}" failed. No retries permitted until 2026-07-13 19:22:45.532041153 +0000 UTC m=+1911.772341230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls") pod "error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" (UID: "5cf5f081-81e8-4cd2-a14e-4d235593ae37") : secret "error-404-isvc-4dc86-predictor-serving-cert" not found
Jul 13 19:22:45.032565 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.032542    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:45.040351 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.040328    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l65w\" (UniqueName: \"kubernetes.io/projected/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-api-access-5l65w\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:45.229459 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.229383    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:45.435629 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.435599    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:45.438056 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.438034    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls\") pod \"success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") " pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:45.536393 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.536358    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:45.538717 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.538700    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls\") pod \"error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:45.654499 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.654457    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:45.763676 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.763635    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:45.789417 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.789390    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"]
Jul 13 19:22:45.910485 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.910158    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"]
Jul 13 19:22:45.914114 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:22:45.914084    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf5f081_81e8_4cd2_a14e_4d235593ae37.slice/crio-e5843104eb623eca2d39a740cfe93bbb03fbe2cd00c8cc10bddd1f49bd229011 WatchSource:0}: Error finding container e5843104eb623eca2d39a740cfe93bbb03fbe2cd00c8cc10bddd1f49bd229011: Status 404 returned error can't find the container with id e5843104eb623eca2d39a740cfe93bbb03fbe2cd00c8cc10bddd1f49bd229011
Jul 13 19:22:45.989018 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.988989    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" event={"ID":"5cf5f081-81e8-4cd2-a14e-4d235593ae37","Type":"ContainerStarted","Data":"6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e"}
Jul 13 19:22:45.989151 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.989054    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" event={"ID":"5cf5f081-81e8-4cd2-a14e-4d235593ae37","Type":"ContainerStarted","Data":"e5843104eb623eca2d39a740cfe93bbb03fbe2cd00c8cc10bddd1f49bd229011"}
Jul 13 19:22:45.990661 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.990640    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" event={"ID":"c0b288cb-227a-4c81-940b-a72e93e2405b","Type":"ContainerStarted","Data":"a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0"}
Jul 13 19:22:45.990755 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.990692    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" event={"ID":"c0b288cb-227a-4c81-940b-a72e93e2405b","Type":"ContainerStarted","Data":"853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025"}
Jul 13 19:22:45.990755 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.990719    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:45.990755 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.990741    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" event={"ID":"c0b288cb-227a-4c81-940b-a72e93e2405b","Type":"ContainerStarted","Data":"c519df6530c94526859b1bc91e0b6e51d6f01d8bd1fc3f352363693fdba0b335"}
Jul 13 19:22:45.990864 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.990769    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:45.992134 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:45.992105    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 19:22:46.012001 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:46.011140    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podStartSLOduration=2.011117782 podStartE2EDuration="2.011117782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:22:46.008670273 +0000 UTC m=+1912.248970384" watchObservedRunningTime="2026-07-13 19:22:46.011117782 +0000 UTC m=+1912.251417884"
Jul 13 19:22:46.846715 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:46.846672    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 19:22:46.996820 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:46.996778    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" event={"ID":"5cf5f081-81e8-4cd2-a14e-4d235593ae37","Type":"ContainerStarted","Data":"b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200"}
Jul 13 19:22:46.997097 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:46.997057    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:46.997252 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:46.997117    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:46.997321 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:46.997264    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 19:22:46.998354 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:46.998327    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 13 19:22:47.018268 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:47.018194    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podStartSLOduration=3.018178482 podStartE2EDuration="3.018178482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:22:47.016680881 +0000 UTC m=+1913.256980982" watchObservedRunningTime="2026-07-13 19:22:47.018178482 +0000 UTC m=+1913.258478581"
Jul 13 19:22:47.771051 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:47.771005    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 13 19:22:47.771274 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:47.771009    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 13 19:22:47.775410 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:47.775386    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 19:22:47.775481 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:47.775386    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 19:22:47.856539 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:47.856496    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 19:22:48.001188 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:48.001150    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 13 19:22:48.942682 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:48.942649    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:22:48.986257 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:48.986195    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:22:49.009963 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.009923    2587 generic.go:356] "Generic (PLEG): container finished" podID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerID="bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957" exitCode=0
Jul 13 19:22:49.010147 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.010119    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"
Jul 13 19:22:49.010514 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.010487    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" event={"ID":"25ec8725-5321-47b2-ab02-605a24f0fdca","Type":"ContainerDied","Data":"bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957"}
Jul 13 19:22:49.010609 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.010555    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8" event={"ID":"25ec8725-5321-47b2-ab02-605a24f0fdca","Type":"ContainerDied","Data":"fad3749a582ecc56fd89d2397f14229c7e10326ad39a381357bdf8cd6c5bf5ba"}
Jul 13 19:22:49.010609 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.010596    2587 scope.go:117] "RemoveContainer" containerID="d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5"
Jul 13 19:22:49.012852 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.012821    2587 generic.go:356] "Generic (PLEG): container finished" podID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerID="8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5" exitCode=0
Jul 13 19:22:49.012984 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.012871    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"
Jul 13 19:22:49.012984 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.012895    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" event={"ID":"a9e26c55-973f-41d1-bfb5-c1af71938dff","Type":"ContainerDied","Data":"8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5"}
Jul 13 19:22:49.012984 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.012950    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm" event={"ID":"a9e26c55-973f-41d1-bfb5-c1af71938dff","Type":"ContainerDied","Data":"5ea2989d8ea1b20596c633efaefcdf3511498b37a361346150e5cd5c04ac46b1"}
Jul 13 19:22:49.019251 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.019231    2587 scope.go:117] "RemoveContainer" containerID="bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957"
Jul 13 19:22:49.028720 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.028699    2587 scope.go:117] "RemoveContainer" containerID="d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5"
Jul 13 19:22:49.029000 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:49.028981    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5\": container with ID starting with d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5 not found: ID does not exist" containerID="d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5"
Jul 13 19:22:49.029057 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.029012    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5"} err="failed to get container status \"d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5\": rpc error: code = NotFound desc = could not find container \"d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5\": container with ID starting with d79c46535af5887d1773ee95931413acc073bb88a64e1190e40b5b768d0de0e5 not found: ID does not exist"
Jul 13 19:22:49.029057 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.029032    2587 scope.go:117] "RemoveContainer" containerID="bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957"
Jul 13 19:22:49.029314 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:49.029295    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957\": container with ID starting with bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957 not found: ID does not exist" containerID="bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957"
Jul 13 19:22:49.029384 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.029323    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957"} err="failed to get container status \"bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957\": rpc error: code = NotFound desc = could not find container \"bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957\": container with ID starting with bd367d3cdedacac340f61d0e00d2c3af53ca0e19b8e4c2eb74a8d75f03d0a957 not found: ID does not exist"
Jul 13 19:22:49.029384 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.029348    2587 scope.go:117] "RemoveContainer" containerID="1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685"
Jul 13 19:22:49.036545 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.036526    2587 scope.go:117] "RemoveContainer" containerID="8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5"
Jul 13 19:22:49.043699 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.043677    2587 scope.go:117] "RemoveContainer" containerID="1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685"
Jul 13 19:22:49.044033 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:49.044015    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685\": container with ID starting with 1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685 not found: ID does not exist" containerID="1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685"
Jul 13 19:22:49.044113 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.044042    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685"} err="failed to get container status \"1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685\": rpc error: code = NotFound desc = could not find container \"1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685\": container with ID starting with 1ab7b0c021a576bcb76ee5bb0021429ea90fe266cffe3622755bdc1503e53685 not found: ID does not exist"
Jul 13 19:22:49.044113 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.044067    2587 scope.go:117] "RemoveContainer" containerID="8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5"
Jul 13 19:22:49.044346 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:22:49.044328    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5\": container with ID starting with 8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5 not found: ID does not exist" containerID="8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5"
Jul 13 19:22:49.044410 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.044355    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5"} err="failed to get container status \"8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5\": rpc error: code = NotFound desc = could not find container \"8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5\": container with ID starting with 8fdaf0f80772a08a820bac04cf9c5c64156aaa5db2d850032b4fc241bcd2ffb5 not found: ID does not exist"
Jul 13 19:22:49.067441 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.067409    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w94g\" (UniqueName: \"kubernetes.io/projected/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-api-access-9w94g\") pod \"25ec8725-5321-47b2-ab02-605a24f0fdca\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") "
Jul 13 19:22:49.067625 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.067449    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-rbac-proxy-sar-config\") pod \"a9e26c55-973f-41d1-bfb5-c1af71938dff\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") "
Jul 13 19:22:49.067625 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.067490    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpnpq\" (UniqueName: \"kubernetes.io/projected/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-api-access-zpnpq\") pod \"a9e26c55-973f-41d1-bfb5-c1af71938dff\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") "
Jul 13 19:22:49.067625 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.067516    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-rbac-proxy-sar-config\") pod \"25ec8725-5321-47b2-ab02-605a24f0fdca\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") "
Jul 13 19:22:49.067625 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.067541    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ec8725-5321-47b2-ab02-605a24f0fdca-proxy-tls\") pod \"25ec8725-5321-47b2-ab02-605a24f0fdca\" (UID: \"25ec8725-5321-47b2-ab02-605a24f0fdca\") "
Jul 13 19:22:49.067625 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.067577    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e26c55-973f-41d1-bfb5-c1af71938dff-proxy-tls\") pod \"a9e26c55-973f-41d1-bfb5-c1af71938dff\" (UID: \"a9e26c55-973f-41d1-bfb5-c1af71938dff\") "
Jul 13 19:22:49.067939 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.067907    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9e26c55-973f-41d1-bfb5-c1af71938dff" (UID: "a9e26c55-973f-41d1-bfb5-c1af71938dff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:22:49.068055 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.067906    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25ec8725-5321-47b2-ab02-605a24f0fdca" (UID: "25ec8725-5321-47b2-ab02-605a24f0fdca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:22:49.069738 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.069708    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ec8725-5321-47b2-ab02-605a24f0fdca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25ec8725-5321-47b2-ab02-605a24f0fdca" (UID: "25ec8725-5321-47b2-ab02-605a24f0fdca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:22:49.069825 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.069713    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-api-access-zpnpq" (OuterVolumeSpecName: "kube-api-access-zpnpq") pod "a9e26c55-973f-41d1-bfb5-c1af71938dff" (UID: "a9e26c55-973f-41d1-bfb5-c1af71938dff"). InnerVolumeSpecName "kube-api-access-zpnpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:49.069825 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.069784    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e26c55-973f-41d1-bfb5-c1af71938dff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9e26c55-973f-41d1-bfb5-c1af71938dff" (UID: "a9e26c55-973f-41d1-bfb5-c1af71938dff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:22:49.070050 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.070028    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-api-access-9w94g" (OuterVolumeSpecName: "kube-api-access-9w94g") pod "25ec8725-5321-47b2-ab02-605a24f0fdca" (UID: "25ec8725-5321-47b2-ab02-605a24f0fdca"). InnerVolumeSpecName "kube-api-access-9w94g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:49.169011 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.168982    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w94g\" (UniqueName: \"kubernetes.io/projected/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-api-access-9w94g\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:49.169011 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.169008    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:49.169011 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.169018    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpnpq\" (UniqueName: \"kubernetes.io/projected/a9e26c55-973f-41d1-bfb5-c1af71938dff-kube-api-access-zpnpq\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:49.169266 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.169027    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ec8725-5321-47b2-ab02-605a24f0fdca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:49.169266 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.169038    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ec8725-5321-47b2-ab02-605a24f0fdca-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:49.169266 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.169047    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e26c55-973f-41d1-bfb5-c1af71938dff-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:22:49.334430 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.334399    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"]
Jul 13 19:22:49.341225 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.341185    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a68dc-predictor-654cb8897d-4l8c8"]
Jul 13 19:22:49.354303 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.354279    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"]
Jul 13 19:22:49.358278 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:49.358256    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a68dc-predictor-8656958995-dh6rm"]
Jul 13 19:22:50.506673 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:50.506635    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" path="/var/lib/kubelet/pods/25ec8725-5321-47b2-ab02-605a24f0fdca/volumes"
Jul 13 19:22:50.507303 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:50.507279    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" path="/var/lib/kubelet/pods/a9e26c55-973f-41d1-bfb5-c1af71938dff/volumes"
Jul 13 19:22:52.002676 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:52.002642    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:22:52.003290 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:52.003260    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 19:22:53.007963 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:53.007926    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:22:53.008708 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:53.008666    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 13 19:22:56.846934 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:56.846891    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 19:22:57.856644 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:22:57.856608    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 19:23:02.003995 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:02.003949    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 19:23:03.009158 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:03.009115    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 13 19:23:06.847048 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:06.847011    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:23:07.857350 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:07.857322    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:23:12.003541 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:12.003495    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 19:23:13.009020 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:13.008973    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 13 19:23:18.975430 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:18.975340    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:19.189797 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:19.189757    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:22.003982 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:22.003924    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 19:23:23.009141 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:23.009102    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 13 19:23:29.062174 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.062106    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:29.156257 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.156217    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"]
Jul 13 19:23:29.156565 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.156535    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container" containerID="cri-o://5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf" gracePeriod=30
Jul 13 19:23:29.156692 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.156575    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kube-rbac-proxy" containerID="cri-o://a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe" gracePeriod=30
Jul 13 19:23:29.186498 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.186463    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:29.208426 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.208391    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"]
Jul 13 19:23:29.208734 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.208704    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container" containerID="cri-o://e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28" gracePeriod=30
Jul 13 19:23:29.208830 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.208743    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kube-rbac-proxy" containerID="cri-o://e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f" gracePeriod=30
Jul 13 19:23:29.231756 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.231729    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"]
Jul 13 19:23:29.232045 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232034    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kube-rbac-proxy"
Jul 13 19:23:29.232084 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232047    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kube-rbac-proxy"
Jul 13 19:23:29.232084 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232064    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kube-rbac-proxy"
Jul 13 19:23:29.232084 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232070    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kube-rbac-proxy"
Jul 13 19:23:29.232084 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232076    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container"
Jul 13 19:23:29.232084 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232082    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container"
Jul 13 19:23:29.232258 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232088    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container"
Jul 13 19:23:29.232258 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232093    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container"
Jul 13 19:23:29.232258 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232147    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kube-rbac-proxy"
Jul 13 19:23:29.232258 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232157    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kube-rbac-proxy"
Jul 13 19:23:29.232258 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232163    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e26c55-973f-41d1-bfb5-c1af71938dff" containerName="kserve-container"
Jul 13 19:23:29.232258 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.232169    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ec8725-5321-47b2-ab02-605a24f0fdca" containerName="kserve-container"
Jul 13 19:23:29.235303 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.235271    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.237837 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.237815    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-08823-predictor-serving-cert\""
Jul 13 19:23:29.237964 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.237820    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-08823-kube-rbac-proxy-sar-config\""
Jul 13 19:23:29.250254 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.250225    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"]
Jul 13 19:23:29.265380 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.265344    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:29.294457 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.294425    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.294637 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.294458    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9zfl\" (UniqueName: \"kubernetes.io/projected/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-api-access-c9zfl\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.294637 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.294495    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.338424 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.338392    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:29.342193 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.342169    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"]
Jul 13 19:23:29.345463 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.345446    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.348146 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.348127    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-08823-kube-rbac-proxy-sar-config\""
Jul 13 19:23:29.348396 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.348377    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-08823-predictor-serving-cert\""
Jul 13 19:23:29.356185 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.356163    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"]
Jul 13 19:23:29.395150 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.395116    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b07f8e20-debd-401e-9683-3962605d94e5-proxy-tls\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.395332 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.395157    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.395332 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.395180    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9zfl\" (UniqueName: \"kubernetes.io/projected/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-api-access-c9zfl\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.395332 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.395238    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.395332 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.395279    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b07f8e20-debd-401e-9683-3962605d94e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.395515 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:23:29.395370    2587 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-08823-predictor-serving-cert: secret "success-200-isvc-08823-predictor-serving-cert" not found
Jul 13 19:23:29.395515 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.395419    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46jdz\" (UniqueName: \"kubernetes.io/projected/b07f8e20-debd-401e-9683-3962605d94e5-kube-api-access-46jdz\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.395515 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:23:29.395441    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls podName:dbf1dcca-102f-4118-abfa-66fc5a1ca204 nodeName:}" failed. No retries permitted until 2026-07-13 19:23:29.895424096 +0000 UTC m=+1956.135724179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls") pod "success-200-isvc-08823-predictor-67685c4f88-qkbp9" (UID: "dbf1dcca-102f-4118-abfa-66fc5a1ca204") : secret "success-200-isvc-08823-predictor-serving-cert" not found
Jul 13 19:23:29.395819 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.395802    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.404797 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.404762    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9zfl\" (UniqueName: \"kubernetes.io/projected/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-api-access-c9zfl\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.496794 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.496760    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b07f8e20-debd-401e-9683-3962605d94e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.496794 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.496798    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jdz\" (UniqueName: \"kubernetes.io/projected/b07f8e20-debd-401e-9683-3962605d94e5-kube-api-access-46jdz\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.497055 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.496913    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b07f8e20-debd-401e-9683-3962605d94e5-proxy-tls\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.497476 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.497453    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b07f8e20-debd-401e-9683-3962605d94e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.499386 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.499360    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b07f8e20-debd-401e-9683-3962605d94e5-proxy-tls\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.507733 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.507681    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jdz\" (UniqueName: \"kubernetes.io/projected/b07f8e20-debd-401e-9683-3962605d94e5-kube-api-access-46jdz\") pod \"error-404-isvc-08823-predictor-698559b985-8l49b\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") " pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.572104 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.572009    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:29.656191 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.656156    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:29.816390 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.816341    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"]
Jul 13 19:23:29.818644 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:23:29.818614    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07f8e20_debd_401e_9683_3962605d94e5.slice/crio-590012e0d2a2158b6f45e32c7c780f5d338ff993d478db7a08af2de2c56c80c4 WatchSource:0}: Error finding container 590012e0d2a2158b6f45e32c7c780f5d338ff993d478db7a08af2de2c56c80c4: Status 404 returned error can't find the container with id 590012e0d2a2158b6f45e32c7c780f5d338ff993d478db7a08af2de2c56c80c4
Jul 13 19:23:29.900120 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.900093    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.903104 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.903075    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls\") pod \"success-200-isvc-08823-predictor-67685c4f88-qkbp9\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") " pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:29.937186 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:29.937158    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:30.145476 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.145380    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:30.179188 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.179155    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" event={"ID":"b07f8e20-debd-401e-9683-3962605d94e5","Type":"ContainerStarted","Data":"b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc"}
Jul 13 19:23:30.179384 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.179249    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" event={"ID":"b07f8e20-debd-401e-9683-3962605d94e5","Type":"ContainerStarted","Data":"c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089"}
Jul 13 19:23:30.179384 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.179322    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:30.179384 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.179372    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:30.179524 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.179400    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" event={"ID":"b07f8e20-debd-401e-9683-3962605d94e5","Type":"ContainerStarted","Data":"590012e0d2a2158b6f45e32c7c780f5d338ff993d478db7a08af2de2c56c80c4"}
Jul 13 19:23:30.180484 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.180452    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 13 19:23:30.181189 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.181165    2587 generic.go:356] "Generic (PLEG): container finished" podID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerID="a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe" exitCode=2
Jul 13 19:23:30.181299 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.181234    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" event={"ID":"893f2442-98f0-45b6-bccd-17a4f7826dbb","Type":"ContainerDied","Data":"a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe"}
Jul 13 19:23:30.183142 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.183115    2587 generic.go:356] "Generic (PLEG): container finished" podID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerID="e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f" exitCode=2
Jul 13 19:23:30.183247 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.183186    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" event={"ID":"6194efd1-4acd-483c-9c77-2dc1b5630ebe","Type":"ContainerDied","Data":"e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f"}
Jul 13 19:23:30.200068 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.199246    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podStartSLOduration=1.199225034 podStartE2EDuration="1.199225034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:23:30.197688387 +0000 UTC m=+1956.437988488" watchObservedRunningTime="2026-07-13 19:23:30.199225034 +0000 UTC m=+1956.439525130"
Jul 13 19:23:30.287129 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:30.287038    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"]
Jul 13 19:23:30.289458 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:23:30.289434    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf1dcca_102f_4118_abfa_66fc5a1ca204.slice/crio-ff0cafb4b6c4848eacc9ab01b3978b07db3093975f532e22fc693f0c1e5c5625 WatchSource:0}: Error finding container ff0cafb4b6c4848eacc9ab01b3978b07db3093975f532e22fc693f0c1e5c5625: Status 404 returned error can't find the container with id ff0cafb4b6c4848eacc9ab01b3978b07db3093975f532e22fc693f0c1e5c5625
Jul 13 19:23:31.189600 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.189567    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" event={"ID":"dbf1dcca-102f-4118-abfa-66fc5a1ca204","Type":"ContainerStarted","Data":"91528244b759be7b553b056546df5159b74d56bc73298dbcc2a3f15528acfaf3"}
Jul 13 19:23:31.190056 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.189626    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" event={"ID":"dbf1dcca-102f-4118-abfa-66fc5a1ca204","Type":"ContainerStarted","Data":"d5f6f4770affbd00621a33db7969225dab85aa287450367819fc4818cb67160e"}
Jul 13 19:23:31.190056 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.189869    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 13 19:23:31.190056 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.189972    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:31.190056 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.190021    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" event={"ID":"dbf1dcca-102f-4118-abfa-66fc5a1ca204","Type":"ContainerStarted","Data":"ff0cafb4b6c4848eacc9ab01b3978b07db3093975f532e22fc693f0c1e5c5625"}
Jul 13 19:23:31.190299 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.190061    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:31.191121 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.191095    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 13 19:23:31.209973 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.209913    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podStartSLOduration=2.20989684 podStartE2EDuration="2.20989684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:23:31.207653525 +0000 UTC m=+1957.447953627" watchObservedRunningTime="2026-07-13 19:23:31.20989684 +0000 UTC m=+1957.450196939"
Jul 13 19:23:31.842124 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:31.842083    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 13 19:23:32.003423 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:32.003381    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 19:23:32.196610 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:32.196514    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 13 19:23:32.851995 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:32.851946    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 13 19:23:33.010334 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.010301    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:23:33.171507 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.171474    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:23:33.202010 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.201970    2587 generic.go:356] "Generic (PLEG): container finished" podID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerID="e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28" exitCode=0
Jul 13 19:23:33.202508 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.202044    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" event={"ID":"6194efd1-4acd-483c-9c77-2dc1b5630ebe","Type":"ContainerDied","Data":"e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28"}
Jul 13 19:23:33.202508 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.202139    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p" event={"ID":"6194efd1-4acd-483c-9c77-2dc1b5630ebe","Type":"ContainerDied","Data":"0da7ca44cc85514738a4ab0ba31d3aa573b53c163c565fb046a4ae9242a1c5e7"}
Jul 13 19:23:33.202508 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.202180    2587 scope.go:117] "RemoveContainer" containerID="e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f"
Jul 13 19:23:33.202684 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.202585    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 13 19:23:33.202822 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.202801    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"
Jul 13 19:23:33.212454 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.212430    2587 scope.go:117] "RemoveContainer" containerID="e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28"
Jul 13 19:23:33.220797 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.220772    2587 scope.go:117] "RemoveContainer" containerID="e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f"
Jul 13 19:23:33.221120 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:23:33.221094    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f\": container with ID starting with e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f not found: ID does not exist" containerID="e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f"
Jul 13 19:23:33.221170 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.221131    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f"} err="failed to get container status \"e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f\": rpc error: code = NotFound desc = could not find container \"e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f\": container with ID starting with e055299509a0a14cad1194a559a6029e75b420f80ef0b79529b4be946686e59f not found: ID does not exist"
Jul 13 19:23:33.221170 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.221150    2587 scope.go:117] "RemoveContainer" containerID="e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28"
Jul 13 19:23:33.221453 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:23:33.221427    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28\": container with ID starting with e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28 not found: ID does not exist" containerID="e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28"
Jul 13 19:23:33.221522 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.221462    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28"} err="failed to get container status \"e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28\": rpc error: code = NotFound desc = could not find container \"e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28\": container with ID starting with e49046ade587d6f5d67de5a16b2145c849c931ec959d255b841e382eefe49b28 not found: ID does not exist"
Jul 13 19:23:33.229734 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.229703    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lkjr\" (UniqueName: \"kubernetes.io/projected/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-api-access-5lkjr\") pod \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") "
Jul 13 19:23:33.229889 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.229748    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-rbac-proxy-sar-config\") pod \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") "
Jul 13 19:23:33.229889 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.229814    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls\") pod \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\" (UID: \"6194efd1-4acd-483c-9c77-2dc1b5630ebe\") "
Jul 13 19:23:33.230250 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.230217    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6194efd1-4acd-483c-9c77-2dc1b5630ebe" (UID: "6194efd1-4acd-483c-9c77-2dc1b5630ebe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:23:33.231936 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.231901    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-api-access-5lkjr" (OuterVolumeSpecName: "kube-api-access-5lkjr") pod "6194efd1-4acd-483c-9c77-2dc1b5630ebe" (UID: "6194efd1-4acd-483c-9c77-2dc1b5630ebe"). InnerVolumeSpecName "kube-api-access-5lkjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:23:33.232021 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.231978    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6194efd1-4acd-483c-9c77-2dc1b5630ebe" (UID: "6194efd1-4acd-483c-9c77-2dc1b5630ebe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:23:33.330396 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.330360    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6194efd1-4acd-483c-9c77-2dc1b5630ebe-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:23:33.330396 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.330390    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lkjr\" (UniqueName: \"kubernetes.io/projected/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-api-access-5lkjr\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:23:33.330396 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.330403    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6194efd1-4acd-483c-9c77-2dc1b5630ebe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:23:33.526503 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.526476    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"]
Jul 13 19:23:33.532124 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:33.532094    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5afb-predictor-6754b9b7-xxv6p"]
Jul 13 19:23:34.505700 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:34.505664    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" path="/var/lib/kubelet/pods/6194efd1-4acd-483c-9c77-2dc1b5630ebe/volumes"
Jul 13 19:23:36.194711 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.194683    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:23:36.195163 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.195127    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 13 19:23:36.412636 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.412605    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:23:36.455569 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.455487    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-rbac-proxy-sar-config\") pod \"893f2442-98f0-45b6-bccd-17a4f7826dbb\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") "
Jul 13 19:23:36.455569 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.455530    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls\") pod \"893f2442-98f0-45b6-bccd-17a4f7826dbb\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") "
Jul 13 19:23:36.455765 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.455571    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtwvz\" (UniqueName: \"kubernetes.io/projected/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-api-access-vtwvz\") pod \"893f2442-98f0-45b6-bccd-17a4f7826dbb\" (UID: \"893f2442-98f0-45b6-bccd-17a4f7826dbb\") "
Jul 13 19:23:36.455906 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.455885    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "893f2442-98f0-45b6-bccd-17a4f7826dbb" (UID: "893f2442-98f0-45b6-bccd-17a4f7826dbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:23:36.457604 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.457579    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "893f2442-98f0-45b6-bccd-17a4f7826dbb" (UID: "893f2442-98f0-45b6-bccd-17a4f7826dbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:23:36.457703 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.457666    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-api-access-vtwvz" (OuterVolumeSpecName: "kube-api-access-vtwvz") pod "893f2442-98f0-45b6-bccd-17a4f7826dbb" (UID: "893f2442-98f0-45b6-bccd-17a4f7826dbb"). InnerVolumeSpecName "kube-api-access-vtwvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:23:36.557224 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.557168    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:23:36.557224 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.557197    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893f2442-98f0-45b6-bccd-17a4f7826dbb-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:23:36.557224 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:36.557229    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwvz\" (UniqueName: \"kubernetes.io/projected/893f2442-98f0-45b6-bccd-17a4f7826dbb-kube-api-access-vtwvz\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:23:37.224696 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.224663    2587 generic.go:356] "Generic (PLEG): container finished" podID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerID="5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf" exitCode=0
Jul 13 19:23:37.225133 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.224713    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"
Jul 13 19:23:37.225133 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.224714    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" event={"ID":"893f2442-98f0-45b6-bccd-17a4f7826dbb","Type":"ContainerDied","Data":"5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf"}
Jul 13 19:23:37.225133 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.224851    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4" event={"ID":"893f2442-98f0-45b6-bccd-17a4f7826dbb","Type":"ContainerDied","Data":"5b0a12847e18c67595d20d8d93171c956c13fbd8f356655028ff6ba034125c41"}
Jul 13 19:23:37.225133 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.224884    2587 scope.go:117] "RemoveContainer" containerID="a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe"
Jul 13 19:23:37.232870 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.232851    2587 scope.go:117] "RemoveContainer" containerID="5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf"
Jul 13 19:23:37.240121 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.240103    2587 scope.go:117] "RemoveContainer" containerID="a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe"
Jul 13 19:23:37.240430 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:23:37.240413    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe\": container with ID starting with a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe not found: ID does not exist" containerID="a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe"
Jul 13 19:23:37.240479 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.240437    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe"} err="failed to get container status \"a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe\": rpc error: code = NotFound desc = could not find container \"a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe\": container with ID starting with a6b6ae38806a74999b4aab30053a966432753fed8ad814062c9506735320fcfe not found: ID does not exist"
Jul 13 19:23:37.240479 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.240454    2587 scope.go:117] "RemoveContainer" containerID="5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf"
Jul 13 19:23:37.240700 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:23:37.240679    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf\": container with ID starting with 5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf not found: ID does not exist" containerID="5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf"
Jul 13 19:23:37.240734 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.240713    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf"} err="failed to get container status \"5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf\": rpc error: code = NotFound desc = could not find container \"5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf\": container with ID starting with 5ba1318a02b5b80f70e9a4febf003eb194ed76e3f9b1a71e6023cc8f3451d4cf not found: ID does not exist"
Jul 13 19:23:37.245209 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.245178    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"]
Jul 13 19:23:37.250558 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:37.250537    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5afb-predictor-5d6dcc6dc4-wgkz4"]
Jul 13 19:23:38.209721 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:38.209675    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:23:38.211296 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:38.211247    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 13 19:23:38.505964 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:38.505878    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" path="/var/lib/kubelet/pods/893f2442-98f0-45b6-bccd-17a4f7826dbb/volumes"
Jul 13 19:23:42.004350 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:42.004320    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:23:46.195755 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:46.195670    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 13 19:23:48.211335 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:48.211292    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 13 19:23:54.828072 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:54.827981    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:55.126806 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:55.126726    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:23:56.195456 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:56.195417    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 13 19:23:58.211371 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:23:58.211331    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 13 19:24:06.195644 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:24:06.195595    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 13 19:24:08.211080 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:24:08.211030    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 13 19:24:16.195414 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:24:16.195374    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 13 19:24:18.210749 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:24:18.210703    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 13 19:24:26.196160 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:24:26.196123    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:24:28.212397 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:24:28.212360    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:24:39.317635 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:24:39.317597    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:24:40.550650 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:24:40.550618    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:25:54.451334 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:25:54.451302    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:25:54.454689 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:25:54.454665    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:27:55.767795 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:27:55.767758    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:29:48.543689 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:29:48.543612    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:30:54.476030 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:30:54.475996    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:30:54.479161 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:30:54.479136    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:31:13.549824 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:31:13.549784    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:31:58.181045 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:31:58.181010    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:09.344911 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:09.344878    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:53.979072 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:53.979029    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:54.075873 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.075837    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"]
Jul 13 19:32:54.076166 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.076139    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container" containerID="cri-o://d5f6f4770affbd00621a33db7969225dab85aa287450367819fc4818cb67160e" gracePeriod=30
Jul 13 19:32:54.076299 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.076222    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kube-rbac-proxy" containerID="cri-o://91528244b759be7b553b056546df5159b74d56bc73298dbcc2a3f15528acfaf3" gracePeriod=30
Jul 13 19:32:54.105435 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.105396    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:54.112303 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.112270    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"]
Jul 13 19:32:54.112534 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.112507    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" containerID="cri-o://c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089" gracePeriod=30
Jul 13 19:32:54.112612 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.112550    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kube-rbac-proxy" containerID="cri-o://b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc" gracePeriod=30
Jul 13 19:32:54.203994 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.203955    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:54.396309 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.396274    2587 generic.go:356] "Generic (PLEG): container finished" podID="b07f8e20-debd-401e-9683-3962605d94e5" containerID="b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc" exitCode=2
Jul 13 19:32:54.396494 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.396350    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" event={"ID":"b07f8e20-debd-401e-9683-3962605d94e5","Type":"ContainerDied","Data":"b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc"}
Jul 13 19:32:54.397923 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.397905    2587 generic.go:356] "Generic (PLEG): container finished" podID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerID="91528244b759be7b553b056546df5159b74d56bc73298dbcc2a3f15528acfaf3" exitCode=2
Jul 13 19:32:54.398018 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:54.397966    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" event={"ID":"dbf1dcca-102f-4118-abfa-66fc5a1ca204","Type":"ContainerDied","Data":"91528244b759be7b553b056546df5159b74d56bc73298dbcc2a3f15528acfaf3"}
Jul 13 19:32:56.190416 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:56.190371    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 13 19:32:56.195806 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:56.195776    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 13 19:32:57.411189 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.411159    2587 generic.go:356] "Generic (PLEG): container finished" podID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerID="d5f6f4770affbd00621a33db7969225dab85aa287450367819fc4818cb67160e" exitCode=0
Jul 13 19:32:57.411623 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.411245    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" event={"ID":"dbf1dcca-102f-4118-abfa-66fc5a1ca204","Type":"ContainerDied","Data":"d5f6f4770affbd00621a33db7969225dab85aa287450367819fc4818cb67160e"}
Jul 13 19:32:57.411623 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.411308    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9" event={"ID":"dbf1dcca-102f-4118-abfa-66fc5a1ca204","Type":"ContainerDied","Data":"ff0cafb4b6c4848eacc9ab01b3978b07db3093975f532e22fc693f0c1e5c5625"}
Jul 13 19:32:57.411623 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.411328    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff0cafb4b6c4848eacc9ab01b3978b07db3093975f532e22fc693f0c1e5c5625"
Jul 13 19:32:57.420809 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.420783    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:32:57.478444 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.478357    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9zfl\" (UniqueName: \"kubernetes.io/projected/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-api-access-c9zfl\") pod \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") "
Jul 13 19:32:57.478444 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.478413    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls\") pod \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") "
Jul 13 19:32:57.478444 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.478435    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-rbac-proxy-sar-config\") pod \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\" (UID: \"dbf1dcca-102f-4118-abfa-66fc5a1ca204\") "
Jul 13 19:32:57.478864 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.478836    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbf1dcca-102f-4118-abfa-66fc5a1ca204" (UID: "dbf1dcca-102f-4118-abfa-66fc5a1ca204"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:32:57.480523 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.480493    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbf1dcca-102f-4118-abfa-66fc5a1ca204" (UID: "dbf1dcca-102f-4118-abfa-66fc5a1ca204"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:32:57.480642 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.480559    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-api-access-c9zfl" (OuterVolumeSpecName: "kube-api-access-c9zfl") pod "dbf1dcca-102f-4118-abfa-66fc5a1ca204" (UID: "dbf1dcca-102f-4118-abfa-66fc5a1ca204"). InnerVolumeSpecName "kube-api-access-c9zfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:32:57.579117 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.579080    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9zfl\" (UniqueName: \"kubernetes.io/projected/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-api-access-c9zfl\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:32:57.579117 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.579113    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbf1dcca-102f-4118-abfa-66fc5a1ca204-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:32:57.579117 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:57.579123    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbf1dcca-102f-4118-abfa-66fc5a1ca204-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:32:58.415055 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:58.415024    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"
Jul 13 19:32:58.437370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:58.437340    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"]
Jul 13 19:32:58.442110 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:58.442086    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08823-predictor-67685c4f88-qkbp9"]
Jul 13 19:32:58.509782 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:32:58.509740    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" path="/var/lib/kubelet/pods/dbf1dcca-102f-4118-abfa-66fc5a1ca204/volumes"
Jul 13 19:33:00.859017 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:00.858993    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:33:00.902101 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:00.902069    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46jdz\" (UniqueName: \"kubernetes.io/projected/b07f8e20-debd-401e-9683-3962605d94e5-kube-api-access-46jdz\") pod \"b07f8e20-debd-401e-9683-3962605d94e5\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") "
Jul 13 19:33:00.902282 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:00.902108    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b07f8e20-debd-401e-9683-3962605d94e5-kube-rbac-proxy-sar-config\") pod \"b07f8e20-debd-401e-9683-3962605d94e5\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") "
Jul 13 19:33:00.902282 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:00.902169    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b07f8e20-debd-401e-9683-3962605d94e5-proxy-tls\") pod \"b07f8e20-debd-401e-9683-3962605d94e5\" (UID: \"b07f8e20-debd-401e-9683-3962605d94e5\") "
Jul 13 19:33:00.902520 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:00.902492    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07f8e20-debd-401e-9683-3962605d94e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b07f8e20-debd-401e-9683-3962605d94e5" (UID: "b07f8e20-debd-401e-9683-3962605d94e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:33:00.904172 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:00.904148    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07f8e20-debd-401e-9683-3962605d94e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b07f8e20-debd-401e-9683-3962605d94e5" (UID: "b07f8e20-debd-401e-9683-3962605d94e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:33:00.904172 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:00.904152    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07f8e20-debd-401e-9683-3962605d94e5-kube-api-access-46jdz" (OuterVolumeSpecName: "kube-api-access-46jdz") pod "b07f8e20-debd-401e-9683-3962605d94e5" (UID: "b07f8e20-debd-401e-9683-3962605d94e5"). InnerVolumeSpecName "kube-api-access-46jdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:33:01.003353 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.003258    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b07f8e20-debd-401e-9683-3962605d94e5-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:33:01.003353 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.003293    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46jdz\" (UniqueName: \"kubernetes.io/projected/b07f8e20-debd-401e-9683-3962605d94e5-kube-api-access-46jdz\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:33:01.003353 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.003307    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b07f8e20-debd-401e-9683-3962605d94e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:33:01.428043 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.428007    2587 generic.go:356] "Generic (PLEG): container finished" podID="b07f8e20-debd-401e-9683-3962605d94e5" containerID="c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089" exitCode=0
Jul 13 19:33:01.428191 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.428054    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"
Jul 13 19:33:01.428191 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.428091    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" event={"ID":"b07f8e20-debd-401e-9683-3962605d94e5","Type":"ContainerDied","Data":"c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089"}
Jul 13 19:33:01.428191 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.428151    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b" event={"ID":"b07f8e20-debd-401e-9683-3962605d94e5","Type":"ContainerDied","Data":"590012e0d2a2158b6f45e32c7c780f5d338ff993d478db7a08af2de2c56c80c4"}
Jul 13 19:33:01.428191 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.428177    2587 scope.go:117] "RemoveContainer" containerID="b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc"
Jul 13 19:33:01.437720 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.437703    2587 scope.go:117] "RemoveContainer" containerID="c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089"
Jul 13 19:33:01.447839 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.447818    2587 scope.go:117] "RemoveContainer" containerID="b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc"
Jul 13 19:33:01.448101 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:33:01.448082    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc\": container with ID starting with b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc not found: ID does not exist" containerID="b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc"
Jul 13 19:33:01.448216 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.448106    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc"} err="failed to get container status \"b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc\": rpc error: code = NotFound desc = could not find container \"b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc\": container with ID starting with b58122726835a6c0e0f97356259d89dac29bb2b506c49bfc272c9396241284bc not found: ID does not exist"
Jul 13 19:33:01.448216 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.448124    2587 scope.go:117] "RemoveContainer" containerID="c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089"
Jul 13 19:33:01.448425 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:33:01.448406    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089\": container with ID starting with c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089 not found: ID does not exist" containerID="c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089"
Jul 13 19:33:01.448466 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.448433    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089"} err="failed to get container status \"c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089\": rpc error: code = NotFound desc = could not find container \"c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089\": container with ID starting with c538ffe89c06f0173b5e99555ad251ec2de7f4b48061a3aa5910a70afe70a089 not found: ID does not exist"
Jul 13 19:33:01.451688 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.451661    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"]
Jul 13 19:33:01.459622 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:01.459600    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08823-predictor-698559b985-8l49b"]
Jul 13 19:33:02.505329 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:02.505295    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b07f8e20-debd-401e-9683-3962605d94e5" path="/var/lib/kubelet/pods/b07f8e20-debd-401e-9683-3962605d94e5/volumes"
Jul 13 19:33:02.948311 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:02.948279    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:54.615070 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:54.615038    2587 scope.go:117] "RemoveContainer" containerID="91528244b759be7b553b056546df5159b74d56bc73298dbcc2a3f15528acfaf3"
Jul 13 19:33:54.623005 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:33:54.622983    2587 scope.go:117] "RemoveContainer" containerID="d5f6f4770affbd00621a33db7969225dab85aa287450367819fc4818cb67160e"
Jul 13 19:35:54.499291 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:35:54.499218    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:35:54.503619 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:35:54.503585    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:40:06.674339 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:06.674284    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:14.248432 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.248393    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:14.370194 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.370162    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"]
Jul 13 19:40:14.370483 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.370448    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container" containerID="cri-o://853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025" gracePeriod=30
Jul 13 19:40:14.370632 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.370502    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kube-rbac-proxy" containerID="cri-o://a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0" gracePeriod=30
Jul 13 19:40:14.393135 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.393100    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:14.420922 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.420884    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"]
Jul 13 19:40:14.421195 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.421167    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container" containerID="cri-o://6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e" gracePeriod=30
Jul 13 19:40:14.421286 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.421237    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kube-rbac-proxy" containerID="cri-o://b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200" gracePeriod=30
Jul 13 19:40:14.451261 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.451213    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:14.947508 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:14.947419    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:15.044194 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:15.044152    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:15.055786 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:15.055748    2587 generic.go:356] "Generic (PLEG): container finished" podID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerID="a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0" exitCode=2
Jul 13 19:40:15.055962 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:15.055814    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" event={"ID":"c0b288cb-227a-4c81-940b-a72e93e2405b","Type":"ContainerDied","Data":"a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0"}
Jul 13 19:40:15.057511 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:15.057488    2587 generic.go:356] "Generic (PLEG): container finished" podID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerID="b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200" exitCode=2
Jul 13 19:40:15.057641 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:15.057538    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" event={"ID":"5cf5f081-81e8-4cd2-a14e-4d235593ae37","Type":"ContainerDied","Data":"b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200"}
Jul 13 19:40:16.997529 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:16.997432    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 13 19:40:17.535653 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.535623    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:40:17.576389 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.576357    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:40:17.627948 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.627902    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls\") pod \"c0b288cb-227a-4c81-940b-a72e93e2405b\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") "
Jul 13 19:40:17.628135 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.627984    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls\") pod \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") "
Jul 13 19:40:17.628135 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.628011    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l65w\" (UniqueName: \"kubernetes.io/projected/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-api-access-5l65w\") pod \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") "
Jul 13 19:40:17.628135 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.628038    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-rbac-proxy-sar-config\") pod \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\" (UID: \"5cf5f081-81e8-4cd2-a14e-4d235593ae37\") "
Jul 13 19:40:17.628135 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.628066    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l6bn\" (UniqueName: \"kubernetes.io/projected/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-api-access-4l6bn\") pod \"c0b288cb-227a-4c81-940b-a72e93e2405b\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") "
Jul 13 19:40:17.628370 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.628248    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-rbac-proxy-sar-config\") pod \"c0b288cb-227a-4c81-940b-a72e93e2405b\" (UID: \"c0b288cb-227a-4c81-940b-a72e93e2405b\") "
Jul 13 19:40:17.628541 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.628512    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cf5f081-81e8-4cd2-a14e-4d235593ae37" (UID: "5cf5f081-81e8-4cd2-a14e-4d235593ae37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:40:17.628672 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.628565    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0b288cb-227a-4c81-940b-a72e93e2405b" (UID: "c0b288cb-227a-4c81-940b-a72e93e2405b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:40:17.630785 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.630755    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0b288cb-227a-4c81-940b-a72e93e2405b" (UID: "c0b288cb-227a-4c81-940b-a72e93e2405b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:40:17.630785 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.630778    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-api-access-4l6bn" (OuterVolumeSpecName: "kube-api-access-4l6bn") pod "c0b288cb-227a-4c81-940b-a72e93e2405b" (UID: "c0b288cb-227a-4c81-940b-a72e93e2405b"). InnerVolumeSpecName "kube-api-access-4l6bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:40:17.630932 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.630784    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cf5f081-81e8-4cd2-a14e-4d235593ae37" (UID: "5cf5f081-81e8-4cd2-a14e-4d235593ae37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:40:17.630932 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.630775    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-api-access-5l65w" (OuterVolumeSpecName: "kube-api-access-5l65w") pod "5cf5f081-81e8-4cd2-a14e-4d235593ae37" (UID: "5cf5f081-81e8-4cd2-a14e-4d235593ae37"). InnerVolumeSpecName "kube-api-access-5l65w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:40:17.729078 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.729030    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0b288cb-227a-4c81-940b-a72e93e2405b-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:40:17.729078 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.729071    2587 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf5f081-81e8-4cd2-a14e-4d235593ae37-proxy-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:40:17.729078 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.729080    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l65w\" (UniqueName: \"kubernetes.io/projected/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-api-access-5l65w\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:40:17.729078 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.729096    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf5f081-81e8-4cd2-a14e-4d235593ae37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:40:17.729379 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.729106    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l6bn\" (UniqueName: \"kubernetes.io/projected/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-api-access-4l6bn\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:40:17.729379 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:17.729115    2587 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0b288cb-227a-4c81-940b-a72e93e2405b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Jul 13 19:40:18.070047 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.070014    2587 generic.go:356] "Generic (PLEG): container finished" podID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerID="853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025" exitCode=0
Jul 13 19:40:18.070485 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.070064    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"
Jul 13 19:40:18.070485 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.070100    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" event={"ID":"c0b288cb-227a-4c81-940b-a72e93e2405b","Type":"ContainerDied","Data":"853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025"}
Jul 13 19:40:18.070485 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.070155    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j" event={"ID":"c0b288cb-227a-4c81-940b-a72e93e2405b","Type":"ContainerDied","Data":"c519df6530c94526859b1bc91e0b6e51d6f01d8bd1fc3f352363693fdba0b335"}
Jul 13 19:40:18.070485 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.070182    2587 scope.go:117] "RemoveContainer" containerID="a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0"
Jul 13 19:40:18.071796 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.071773    2587 generic.go:356] "Generic (PLEG): container finished" podID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerID="6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e" exitCode=0
Jul 13 19:40:18.071931 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.071817    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"
Jul 13 19:40:18.071931 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.071820    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" event={"ID":"5cf5f081-81e8-4cd2-a14e-4d235593ae37","Type":"ContainerDied","Data":"6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e"}
Jul 13 19:40:18.072052 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.071943    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr" event={"ID":"5cf5f081-81e8-4cd2-a14e-4d235593ae37","Type":"ContainerDied","Data":"e5843104eb623eca2d39a740cfe93bbb03fbe2cd00c8cc10bddd1f49bd229011"}
Jul 13 19:40:18.080978 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.080316    2587 scope.go:117] "RemoveContainer" containerID="853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025"
Jul 13 19:40:18.089251 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.089231    2587 scope.go:117] "RemoveContainer" containerID="a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0"
Jul 13 19:40:18.089537 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:40:18.089514    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0\": container with ID starting with a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0 not found: ID does not exist" containerID="a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0"
Jul 13 19:40:18.089606 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.089545    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0"} err="failed to get container status \"a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0\": rpc error: code = NotFound desc = could not find container \"a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0\": container with ID starting with a38fd0bdc722d7d6cf8ac800d1e508594ec4621310a201cee21632a8b7c3b0a0 not found: ID does not exist"
Jul 13 19:40:18.089606 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.089563    2587 scope.go:117] "RemoveContainer" containerID="853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025"
Jul 13 19:40:18.089808 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:40:18.089787    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025\": container with ID starting with 853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025 not found: ID does not exist" containerID="853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025"
Jul 13 19:40:18.089849 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.089817    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025"} err="failed to get container status \"853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025\": rpc error: code = NotFound desc = could not find container \"853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025\": container with ID starting with 853dfbb440739e6c5004b2126a87449ec061c29591972c36766a5a1549dd6025 not found: ID does not exist"
Jul 13 19:40:18.089849 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.089842    2587 scope.go:117] "RemoveContainer" containerID="b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200"
Jul 13 19:40:18.096722 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.096698    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"]
Jul 13 19:40:18.097553 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.097538    2587 scope.go:117] "RemoveContainer" containerID="6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e"
Jul 13 19:40:18.100267 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.100244    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc86-predictor-79bccf84c7-zpzcr"]
Jul 13 19:40:18.104996 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.104971    2587 scope.go:117] "RemoveContainer" containerID="b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200"
Jul 13 19:40:18.105244 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:40:18.105226    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200\": container with ID starting with b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200 not found: ID does not exist" containerID="b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200"
Jul 13 19:40:18.105296 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.105251    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200"} err="failed to get container status \"b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200\": rpc error: code = NotFound desc = could not find container \"b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200\": container with ID starting with b0bbe2055c3d4f76aac814b9194610ed278e9eb803ef3f4ee5ebd286e2ebf200 not found: ID does not exist"
Jul 13 19:40:18.105296 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.105271    2587 scope.go:117] "RemoveContainer" containerID="6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e"
Jul 13 19:40:18.105508 ip-10-0-139-100 kubenswrapper[2587]: E0713 19:40:18.105489    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e\": container with ID starting with 6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e not found: ID does not exist" containerID="6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e"
Jul 13 19:40:18.105546 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.105514    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e"} err="failed to get container status \"6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e\": rpc error: code = NotFound desc = could not find container \"6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e\": container with ID starting with 6f2b4181b55890731afd678d863931a472f2156fb41431a95691130f39c9f52e not found: ID does not exist"
Jul 13 19:40:18.110841 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.110817    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"]
Jul 13 19:40:18.114595 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.114575    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dc86-predictor-7db9988c5b-ktd6j"]
Jul 13 19:40:18.504927 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.504845    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" path="/var/lib/kubelet/pods/5cf5f081-81e8-4cd2-a14e-4d235593ae37/volumes"
Jul 13 19:40:18.505306 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:18.505292    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" path="/var/lib/kubelet/pods/c0b288cb-227a-4c81-940b-a72e93e2405b/volumes"
Jul 13 19:40:46.073852 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:46.073809    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:46.264622 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:46.264584    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:46.358632 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:46.358535    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:49.000608 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:49.000574    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8gk2r_d7146a5f-6f9b-417d-94f3-8aabf7e61501/global-pull-secret-syncer/0.log"
Jul 13 19:40:49.132117 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:49.132079    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log"
Jul 13 19:40:49.286709 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:49.286681    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-100.ec2.internal_a676b35a3de8ad1194c9799bed61c8c9/haproxy/0.log"
Jul 13 19:40:51.909567 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:51.909519    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:52.011177 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:52.011135    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-94v4l_f1535d30-9a6f-45a5-8f03-f34baaa3ebf4/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:54.297572 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:54.297542    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j59w7_946481d5-3192-42a0-949d-6494bc388aa6/node-exporter/0.log"
Jul 13 19:40:54.316690 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:54.316658    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j59w7_946481d5-3192-42a0-949d-6494bc388aa6/kube-rbac-proxy/0.log"
Jul 13 19:40:54.342531 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:54.342505    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j59w7_946481d5-3192-42a0-949d-6494bc388aa6/init-textfile/0.log"
Jul 13 19:40:54.520626 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:54.520599    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:40:54.525767 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:54.525744    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:40:56.312056 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:56.312018    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-hrpdn_19ecb11a-0f10-49d6-ad5e-ce2f00397c0f/networking-console-plugin/0.log"
Jul 13 19:40:57.115641 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115607    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"]
Jul 13 19:40:57.115883 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115871    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container"
Jul 13 19:40:57.115930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115884    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container"
Jul 13 19:40:57.115930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115898    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container"
Jul 13 19:40:57.115930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115903    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container"
Jul 13 19:40:57.115930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115910    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container"
Jul 13 19:40:57.115930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115916    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container"
Jul 13 19:40:57.115930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115922    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kube-rbac-proxy"
Jul 13 19:40:57.115930 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115927    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115937    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115943    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115949    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115956    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115962    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115967    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115976    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115983    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115991    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.115996    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116003    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116008    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116013    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116018    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116024    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116029    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116073    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116080    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116089    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116095    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116102    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116107    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf1dcca-102f-4118-abfa-66fc5a1ca204" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116112    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf5f081-81e8-4cd2-a14e-4d235593ae37" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116116    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="893f2442-98f0-45b6-bccd-17a4f7826dbb" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116122    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116127    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6194efd1-4acd-483c-9c77-2dc1b5630ebe" containerName="kserve-container"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116132    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b288cb-227a-4c81-940b-a72e93e2405b" containerName="kube-rbac-proxy"
Jul 13 19:40:57.116136 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.116137    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b07f8e20-debd-401e-9683-3962605d94e5" containerName="kserve-container"
Jul 13 19:40:57.120449 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.120430    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.123147 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.123126    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qwzz7\"/\"openshift-service-ca.crt\""
Jul 13 19:40:57.124232 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.124217    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qwzz7\"/\"default-dockercfg-6c7xb\""
Jul 13 19:40:57.124322 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.124282    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qwzz7\"/\"kube-root-ca.crt\""
Jul 13 19:40:57.130221 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.130178    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"]
Jul 13 19:40:57.245227 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.245161    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-proc\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.245419 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.245239    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-lib-modules\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.245419 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.245294    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-sys\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.245419 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.245329    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-podres\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.245419 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.245363    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z47f\" (UniqueName: \"kubernetes.io/projected/08bd7ba6-f694-41ab-9cee-5f4533277b2f-kube-api-access-5z47f\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.346665 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.346629    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-sys\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.347187 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.347161    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-podres\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.347923 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.347623    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-sys\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.347923 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.346666    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-podres\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.347923 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.347833    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z47f\" (UniqueName: \"kubernetes.io/projected/08bd7ba6-f694-41ab-9cee-5f4533277b2f-kube-api-access-5z47f\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.347923 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.347925    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-proc\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.348177 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.347979    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-lib-modules\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.348177 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.348156    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-lib-modules\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.349138 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.348587    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08bd7ba6-f694-41ab-9cee-5f4533277b2f-proc\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.356555 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.356532    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z47f\" (UniqueName: \"kubernetes.io/projected/08bd7ba6-f694-41ab-9cee-5f4533277b2f-kube-api-access-5z47f\") pod \"perf-node-gather-daemonset-xhx26\" (UID: \"08bd7ba6-f694-41ab-9cee-5f4533277b2f\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.430368 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.430270    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:57.556971 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:57.556946    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"]
Jul 13 19:40:57.559561 ip-10-0-139-100 kubenswrapper[2587]: W0713 19:40:57.559539    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod08bd7ba6_f694_41ab_9cee_5f4533277b2f.slice/crio-2aac6e7e74dabbc45c5023b6791aad29e8f6547bb9727248ce59d24091366ebb WatchSource:0}: Error finding container 2aac6e7e74dabbc45c5023b6791aad29e8f6547bb9727248ce59d24091366ebb: Status 404 returned error can't find the container with id 2aac6e7e74dabbc45c5023b6791aad29e8f6547bb9727248ce59d24091366ebb
Jul 13 19:40:58.227268 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:58.227226    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26" event={"ID":"08bd7ba6-f694-41ab-9cee-5f4533277b2f","Type":"ContainerStarted","Data":"99b5ca532fe5bb6f6b0325059bbe91ce36ada120a453bd7fc690789c452bfeeb"}
Jul 13 19:40:58.227453 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:58.227290    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26" event={"ID":"08bd7ba6-f694-41ab-9cee-5f4533277b2f","Type":"ContainerStarted","Data":"2aac6e7e74dabbc45c5023b6791aad29e8f6547bb9727248ce59d24091366ebb"}
Jul 13 19:40:58.227453 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:58.227329    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:40:58.243219 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:58.243154    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26" podStartSLOduration=1.243140202 podStartE2EDuration="1.243140202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:40:58.242149151 +0000 UTC m=+3004.482449266" watchObservedRunningTime="2026-07-13 19:40:58.243140202 +0000 UTC m=+3004.483440302"
Jul 13 19:40:58.684537 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:58.684502    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k475c_9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13/dns/0.log"
Jul 13 19:40:58.704808 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:58.704753    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k475c_9cfe0588-1fa2-4cc4-a5b3-3eeaae8abb13/kube-rbac-proxy/0.log"
Jul 13 19:40:58.806857 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:58.806829    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqc2j_4b227090-d946-4bf0-be0f-cba5485fd90f/dns-node-resolver/0.log"
Jul 13 19:40:59.326077 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:40:59.326029    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8fl4m_29c31923-0274-4781-be82-8266a0459019/node-ca/0.log"
Jul 13 19:41:00.157075 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:00.157043    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c6c7c4594-qv9w2_8ec3b9d5-cd39-418d-8c52-d8cdc7bd10e7/router/0.log"
Jul 13 19:41:00.542211 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:00.542162    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kcvql_66a1f85c-7204-4ef9-a3a4-fe9ae3c636e2/serve-healthcheck-canary/0.log"
Jul 13 19:41:01.106419 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:01.106381    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x8zfs_b7de805b-a750-4359-afc3-61b3e1d2cac7/kube-rbac-proxy/0.log"
Jul 13 19:41:01.126483 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:01.126454    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x8zfs_b7de805b-a750-4359-afc3-61b3e1d2cac7/exporter/0.log"
Jul 13 19:41:01.147383 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:01.147350    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x8zfs_b7de805b-a750-4359-afc3-61b3e1d2cac7/extractor/0.log"
Jul 13 19:41:03.335127 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:03.335085    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6b69b6d547-6g2dl_4b31d809-7dca-44d1-8060-2b7a2be8c41e/manager/0.log"
Jul 13 19:41:03.381805 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:03.381778    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-7n2qz_19803efb-662f-4eb6-9005-e2bdbdecd137/server/0.log"
Jul 13 19:41:03.570088 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:03.570051    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-8pwhq_bc8a59a0-69d3-4dcb-b09c-fe41735f9319/seaweedfs/0.log"
Jul 13 19:41:04.240735 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:04.240706    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-xhx26"
Jul 13 19:41:07.678990 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:07.678960    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-5q57j_2dd5bac6-7970-444d-8238-328f4af61ab2/migrator/0.log"
Jul 13 19:41:07.701565 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:07.701530    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-5q57j_2dd5bac6-7970-444d-8238-328f4af61ab2/graceful-termination/0.log"
Jul 13 19:41:08.950857 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:08.950776    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5z52j_d1f3c2f9-e9bb-4fd4-bdbb-d14f3151d9f4/kube-multus/0.log"
Jul 13 19:41:08.972642 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:08.972618    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhgp5_5bed48f8-1564-4705-9258-755e26ef91cb/kube-multus-additional-cni-plugins/0.log"
Jul 13 19:41:08.995041 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:08.995015    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhgp5_5bed48f8-1564-4705-9258-755e26ef91cb/egress-router-binary-copy/0.log"
Jul 13 19:41:09.013720 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:09.013692    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhgp5_5bed48f8-1564-4705-9258-755e26ef91cb/cni-plugins/0.log"
Jul 13 19:41:09.032563 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:09.032534    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhgp5_5bed48f8-1564-4705-9258-755e26ef91cb/bond-cni-plugin/0.log"
Jul 13 19:41:09.055020 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:09.054987    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhgp5_5bed48f8-1564-4705-9258-755e26ef91cb/routeoverride-cni/0.log"
Jul 13 19:41:09.073766 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:09.073736    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhgp5_5bed48f8-1564-4705-9258-755e26ef91cb/whereabouts-cni-bincopy/0.log"
Jul 13 19:41:09.093695 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:09.093665    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhgp5_5bed48f8-1564-4705-9258-755e26ef91cb/whereabouts-cni/0.log"
Jul 13 19:41:09.604549 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:09.604501    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ll8qw_a21fd6a7-5724-4021-928b-f98404324fb8/network-metrics-daemon/0.log"
Jul 13 19:41:09.621787 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:09.621756    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ll8qw_a21fd6a7-5724-4021-928b-f98404324fb8/kube-rbac-proxy/0.log"
Jul 13 19:41:10.336627 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.336592    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-controller/0.log"
Jul 13 19:41:10.368639 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.368583    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/1.log"
Jul 13 19:41:10.368639 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.368617    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovn-acl-logging/0.log"
Jul 13 19:41:10.388703 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.388672    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/kube-rbac-proxy-node/0.log"
Jul 13 19:41:10.411262 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.411221    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/kube-rbac-proxy-ovn-metrics/0.log"
Jul 13 19:41:10.428753 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.428725    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/northd/0.log"
Jul 13 19:41:10.448103 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.448069    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/nbdb/0.log"
Jul 13 19:41:10.468619 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.468578    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/sbdb/0.log"
Jul 13 19:41:10.532906 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:10.532878    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7fsmf_762f4cb1-637f-40ee-b3ea-2489ccd2f4b9/ovnkube-controller/0.log"
Jul 13 19:41:11.893427 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:11.893390    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-b8h47_92c24328-93b8-4f4a-abdd-dd0312ef081c/check-endpoints/0.log"
Jul 13 19:41:11.967893 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:11.967864    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xvd96_a95bbd20-fbdc-48c9-bed2-2341f164d5e7/network-check-target-container/0.log"
Jul 13 19:41:12.895121 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:12.895092    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-dh5pz_60b983b9-9d4d-42fa-b892-465c5104485f/iptables-alerter/0.log"
Jul 13 19:41:13.577280 ip-10-0-139-100 kubenswrapper[2587]: I0713 19:41:13.577239    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-v5hqb_dff1207e-de9a-45a7-9629-a3e2074a60eb/tuned/0.log"