Jul 10 18:38:06.869120 ip-10-0-129-73 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 10 18:38:06.869131 ip-10-0-129-73 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 10 18:38:06.869138 ip-10-0-129-73 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 10 18:38:06.869346 ip-10-0-129-73 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 10 18:38:17.118773 ip-10-0-129-73 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 10 18:38:17.118794 ip-10-0-129-73 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 372b7635f1be4f8599f22ec32587bca8 --
Jul 10 18:40:18.581233 ip-10-0-129-73 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 18:40:18.989107 ip-10-0-129-73 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:40:18.989107 ip-10-0-129-73 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 18:40:18.989107 ip-10-0-129-73 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:40:18.989107 ip-10-0-129-73 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 18:40:18.989107 ip-10-0-129-73 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:40:18.989941 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.989851    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 18:40:18.995132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995113    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:18.995132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995132    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995136    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995139    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995143    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995146    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995151    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995157    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995160    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995163    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995168    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995172    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995175    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995178    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995181    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995184    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995188    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995192    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995195    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995198    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:18.995203 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995200    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995203    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995205    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995208    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995211    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995214    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995217    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995220    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995223    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995225    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995228    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995231    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995233    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995237    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995239    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995242    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995245    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995247    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995250    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995253    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:18.995640 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995256    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995258    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995261    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995263    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995265    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995268    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995270    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995272    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995275    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995277    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995279    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995282    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995284    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995287    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995290    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995299    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995302    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995305    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995307    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995310    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:18.996130 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995312    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995315    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995318    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995320    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995322    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995325    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995327    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995330    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995332    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995335    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995337    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995340    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995350    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995353    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995356    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995358    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995361    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995363    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995366    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995368    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:18.996605 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995371    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995375    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995377    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995380    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995383    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995385    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995389    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995392    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995395    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995397    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995400    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995402    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995405    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995408    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995410    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995414    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995418    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995420    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.995423    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996173    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:18.997098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996180    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996183    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996186    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996189    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996191    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996194    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996197    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996200    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996203    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996205    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996208    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996211    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996213    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996216    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996218    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996221    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996223    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996226    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996230    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996232    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:18.997590 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996235    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996238    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996240    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996243    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996246    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996248    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996251    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996253    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996256    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996258    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996260    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996263    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996265    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996268    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996270    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996273    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996275    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996278    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996281    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996284    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:18.998132 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996286    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996289    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996291    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996294    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996296    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996299    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996301    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996305    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996309    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996312    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996317    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996321    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996324    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996327    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996330    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996333    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996335    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996338    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996340    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996343    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:18.998601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996345    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996348    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996351    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996353    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996356    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996358    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996361    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996364    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996366    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996369    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996371    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996382    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996384    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996387    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996390    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996394    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996397    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996400    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996404    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:18.999098 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996408    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996412    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996415    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996419    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996422    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996425    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996427    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996430    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996433    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996435    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996438    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996440    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996443    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996445    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996448    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996450    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996453    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996455    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996458    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996882    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:18.999559 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996887    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996890    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996893    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996896    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996899    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996902    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996904    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996907    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996910    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996912    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996915    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996917    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996920    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996922    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996925    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996928    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996931    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996933    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996936    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996938    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:19.000057 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996941    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996943    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996946    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996965    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996969    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996972    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996975    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996978    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996980    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996983    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996985    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996988    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996991    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996993    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.996997    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997002    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997005    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997016    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997019    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:19.000532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997023    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997026    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997030    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997033    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997036    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997039    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997041    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997044    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997047    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997051    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997054    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997057    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997060    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997062    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997065    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997068    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997071    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997074    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997077    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:19.001001 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997080    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997082    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997085    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997088    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997091    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997093    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997096    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997099    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997102    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997105    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997108    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997110    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997113    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997117    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997120    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997122    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997125    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997127    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997131    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997133    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:19.001465 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997135    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997138    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997141    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997143    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997146    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997148    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997151    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997153    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997155    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997158    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997160    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997163    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997165    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997168    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997170    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997173    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997175    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997178    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997181    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997184    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997259    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 18:40:19.002000 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997276    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997283    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997287    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997294    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997298    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997302    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997307    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997310    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997313    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997316    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997320    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997323    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997326    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997329    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997332    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997335    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997338    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997342    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997345    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997348    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997351    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997355    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997359    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 18:40:19.002499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997362    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997366    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997369    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997372    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997377    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997380    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997384    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997386    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997391    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997394    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997396    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997399    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997404    2565 flags.go:64] FLAG: --enable-server="true"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997407    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997415    2565 flags.go:64] FLAG: --event-burst="100"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997419    2565 flags.go:64] FLAG: --event-qps="50"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997422    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997425    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997428    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997431    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997434    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997437    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997440    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997443    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997446    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 18:40:19.003119 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997449    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997452    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997455    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997458    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997461    2565 flags.go:64] FLAG: --feature-gates=""
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997465    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997468    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997471    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997474    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997477    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997480    2565 flags.go:64] FLAG: --help="false"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997485    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997488    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997491    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997494    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997498    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997501    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997504    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997507    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997511    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997514    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997517    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997521    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997524    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 10 18:40:19.003719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997527    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997530    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997532    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997535    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997538    2565 flags.go:64] FLAG: --lock-file=""
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997541    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997544    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997547    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997552    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997555    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997558    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997561    2565 flags.go:64] FLAG: --logging-format="text"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997563    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997567    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997570    2565 flags.go:64] FLAG: --manifest-url=""
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997573    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997577    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997580    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997585    2565 flags.go:64] FLAG: --max-pods="110"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997589    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997592    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997595    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997598    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997601    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997604    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 18:40:19.004362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997607    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997614    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997618    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997621    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997624    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997627    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997634    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997636    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997639    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997642    2565 flags.go:64] FLAG: --port="10250"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997645    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997648    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-048387881b23d4dd8"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997651    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997654    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997657    2565 flags.go:64] FLAG: --register-node="true"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997660    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997664    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997666    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997669    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997672    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997676    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997679    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997682    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997684    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997687    2565 flags.go:64] FLAG: --runonce="false"
Jul 10 18:40:19.005100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997690    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997693    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997698    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997701    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997704    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997707    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997711    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997714    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997716    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997720    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997723    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997726    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997729    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997732    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997735    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997740    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997743    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997746    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997750    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997753    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997756    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997758    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997761    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997764    2565 flags.go:64] FLAG: --v="2"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997768    2565 flags.go:64] FLAG: --version="false"
Jul 10 18:40:19.005989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997772    2565 flags.go:64] FLAG: --vmodule=""
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997775    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.997778    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997917    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997921    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997924    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997927    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997929    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997932    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997936    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997939    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997942    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997944    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997947    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997967    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997972    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997976    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997980    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997985    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997988    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:19.006601 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997990    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997993    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997997    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.997999    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998002    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998005    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998008    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998011    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998013    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998016    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998018    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998021    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998024    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998026    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998029    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998032    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998034    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998036    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998039    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998041    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:19.007133 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998044    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998048    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998050    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998053    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998056    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998058    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998061    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998063    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998066    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998070    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998074    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998076    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998079    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998082    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998084    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998087    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998090    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998092    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998095    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998097    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:19.007599 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998100    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998104    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998108    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998111    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998114    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998117    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998119    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998122    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998124    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998127    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998130    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998132    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998135    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998139    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998142    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998145    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998148    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998152    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998155    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:19.008120 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998158    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998163    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998166    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998169    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998172    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998175    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998177    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998180    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998182    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998186    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998188    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998191    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998193    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998196    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998198    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998201    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998203    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998205    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998208    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998210    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:19.008571 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998213    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998215    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:18.998219    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:18.998849    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.007874    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.007892    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.007908    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.007920    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008003    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008009    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008013    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008015    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008018    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008021    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008024    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:19.009080 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008026    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008029    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008031    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008034    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008037    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008042    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008049    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008054    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008059    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008063    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008066    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008069    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008072    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008075    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008078    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008081    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008083    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008086    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008089    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:19.009443 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008093    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008096    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008098    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008101    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008104    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008107    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008109    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008112    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008131    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008136    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008142    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008147    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008152    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008155    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008158    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008161    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008164    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008167    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008170    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008172    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:19.009882 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008175    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008178    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008181    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008183    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008186    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008188    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008191    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008194    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008197    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008200    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008202    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008206    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008210    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008213    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008216    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008218    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008221    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008224    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008227    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:19.010532 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008229    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008232    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008234    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008237    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008240    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008242    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008244    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008247    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008250    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008252    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008255    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008257    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008260    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008262    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008266    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008268    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008271    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008273    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008278    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008282    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:19.011006 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008287    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008290    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008293    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008295    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008298    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008301    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008304    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008306    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008309    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008313    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008316    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008318    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008321    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008324    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.008329    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:40:19.011476 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008446    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008451    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008454    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008457    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008460    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008463    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008466    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008470    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008473    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008476    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008478    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008481    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008484    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008487    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008490    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008493    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008497    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008504    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008509    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:19.011833 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008513    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008515    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008518    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008520    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008523    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008526    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008528    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008531    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008533    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008536    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008538    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008541    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008544    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008546    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008549    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008551    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008554    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008556    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008558    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008561    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:19.012308 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008563    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008566    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008568    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008571    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008574    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008576    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008579    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008582    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008585    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008587    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008591    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008594    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008598    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008601    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008604    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008607    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008610    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008613    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008616    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:19.012781 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008618    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008621    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008624    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008626    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008629    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008632    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008634    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008636    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008639    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008642    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008644    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008647    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008649    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008652    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008654    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008657    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008660    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008662    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008665    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008667    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:19.013251 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008671    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008673    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008676    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008678    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008681    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008683    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008686    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008688    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008691    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008694    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008696    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008699    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008701    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008704    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008707    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008709    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008711    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008714    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008716    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008719    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:19.013719 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:19.008721    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:19.014199 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.008726    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:40:19.014199 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.008856    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 18:40:19.014199 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.011579    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 18:40:19.014199 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.012914    2565 server.go:1019] "Starting client certificate rotation"
Jul 10 18:40:19.014199 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.013005    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:40:19.014199 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.013056    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:40:19.015475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.015461    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:40:19.021201 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.021181    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:40:19.037801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.037779    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 10 18:40:19.040003 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.039978    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:40:19.040112 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.040098    2565 log.go:25] "Validated CRI v1 image API"
Jul 10 18:40:19.041004 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.040986    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 18:40:19.045530 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.045509    2565 fs.go:135] Filesystem UUIDs: map[765c7fe0-a884-46f1-b3ca-88adc1346a1a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ddd8db1b-6686-4ca4-a5d4-72d3bd9752a0:/dev/nvme0n1p4]
Jul 10 18:40:19.045609 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.045529    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 18:40:19.052932 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.052804    2565 manager.go:216] Machine: {Timestamp:2026-07-10 18:40:19.051354568 +0000 UTC m=+0.349696628 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100935 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21862e00300f1e5eabe1d287dc04fd SystemUUID:ec21862e-0030-0f1e-5eab-e1d287dc04fd BootID:372b7635-f1be-4f85-99f2-2ec32587bca8 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9a:5c:52:10:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9a:5c:52:10:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:98:3a:b5:40:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 18:40:19.052932 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.052922    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 18:40:19.053087 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.053036    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 18:40:19.056262 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.056232    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 18:40:19.056401 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.056266    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-73.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 18:40:19.057025 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.057014    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 18:40:19.057061 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.057027    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 18:40:19.057061 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.057035    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:40:19.057878 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.057868    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:40:19.059205 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.059194    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 18:40:19.061911 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.061898    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:40:19.062251 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.062240    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 18:40:19.062284 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.062276    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 18:40:19.062317 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.062289    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 18:40:19.062317 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.062305    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 18:40:19.062317 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.062314    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 10 18:40:19.062424 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.062323    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 18:40:19.062702 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.062686    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:40:19.062702 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.062706    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:40:19.065275 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.065257    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 18:40:19.066193 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.066176    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jmw72"
Jul 10 18:40:19.067241 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.067224    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 18:40:19.067324 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.067263    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 18:40:19.067464 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.067451    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 18:40:19.067527 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.067471    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 18:40:19.067527 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.067480    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 18:40:19.067527 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.067488    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 18:40:19.067527 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.067501    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 18:40:19.067527 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.067509    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 18:40:19.068050 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.068040    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 18:40:19.068108 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.068052    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 18:40:19.068108 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.068064    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 18:40:19.068108 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.068072    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 18:40:19.068108 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.068085    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 18:40:19.068108 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.068096    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 18:40:19.068862 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.068848    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 18:40:19.068946 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.068865    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 18:40:19.072118 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.072103    2565 server.go:1268] "Started kubelet"
Jul 10 18:40:19.072264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.072238    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 18:40:19.072379 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.072319    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 18:40:19.072434 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.072409    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 18:40:19.073525 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.073496    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 18:40:19.073604 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.073532    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-73.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 10 18:40:19.073666 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.073617    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 18:40:19.073720 ip-10-0-129-73 systemd[1]: Started Kubernetes Kubelet.
Jul 10 18:40:19.073813 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.073785    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 18:40:19.075296 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.075262    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 18:40:19.075751 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.075731    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jmw72"
Jul 10 18:40:19.078371 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.077330    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-73.ec2.internal.18c101b86f85cfb8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-73.ec2.internal,UID:ip-10-0-129-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-73.ec2.internal,},FirstTimestamp:2026-07-10 18:40:19.072077752 +0000 UTC m=+0.370419812,LastTimestamp:2026-07-10 18:40:19.072077752 +0000 UTC m=+0.370419812,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-73.ec2.internal,}"
Jul 10 18:40:19.079355 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.079338    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 18:40:19.080890 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.080873    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 18:40:19.080978 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.080880    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 18:40:19.081050 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.081005    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.082079 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.081947    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 18:40:19.082385 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.082024    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 18:40:19.082468 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.082389    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 18:40:19.082468 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.082418    2565 factory.go:55] Registering systemd factory
Jul 10 18:40:19.082468 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.082434    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 10 18:40:19.082606 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.082471    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 18:40:19.082606 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.082482    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 18:40:19.084096 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.084074    2565 factory.go:153] Registering CRI-O factory
Jul 10 18:40:19.084186 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.084101    2565 factory.go:223] Registration of the crio container factory successfully
Jul 10 18:40:19.084186 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.084157    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 18:40:19.084186 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.084185    2565 factory.go:103] Registering Raw factory
Jul 10 18:40:19.084364 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.084204    2565 manager.go:1196] Started watching for new ooms in manager
Jul 10 18:40:19.084687 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.084672    2565 manager.go:349] Starting recovery of all containers
Jul 10 18:40:19.086915 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.086896    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:19.090889 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.090866    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-73.ec2.internal\" not found" node="ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.096797 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.096659    2565 manager.go:354] Recovery completed
Jul 10 18:40:19.101431 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.101417    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:19.103466 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.103445    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:19.103514 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.103494    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:19.103514 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.103509    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:19.104039 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.104026    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 18:40:19.104096 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.104040    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 18:40:19.104096 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.104061    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:40:19.104096 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.104075    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:19.104764 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.104748    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:19.104829 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.104773    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:19.104829 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.104782    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:19.105273 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.105262    2565 policy_none.go:49] "None policy: Start"
Jul 10 18:40:19.105307 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.105277    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 18:40:19.105307 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.105286    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 18:40:19.106314 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.106304    2565 policy_none.go:47] "Start"
Jul 10 18:40:19.144642 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.144607    2565 manager.go:337] "Starting Device Plugin manager"
Jul 10 18:40:19.144779 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.144673    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 18:40:19.144779 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.144687    2565 server.go:87] "Starting device plugin registration server"
Jul 10 18:40:19.145034 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.145021    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 18:40:19.145106 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.145038    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 18:40:19.145161 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.145146    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 18:40:19.145232 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.145221    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 18:40:19.145279 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.145234    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 18:40:19.145828 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.145795    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 18:40:19.145902 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.145852    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.245206 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.245117    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:19.246386 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.246370    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:19.246473 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.246406    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:19.246473 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.246420    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:19.246570 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.246476    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.255391 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.255369    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.255464 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.255394    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-73.ec2.internal\": node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.266125 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.266100    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 18:40:19.267389 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.267365    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 18:40:19.267509 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.267402    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 18:40:19.267509 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.267423    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 18:40:19.267509 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.267455    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 18:40:19.269705 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.269675    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:19.272329 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.272313    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.367628 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.367589    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal"]
Jul 10 18:40:19.367751 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.367675    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:19.370878 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.370857    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:19.370999 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.370889    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:19.370999 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.370900    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:19.372286 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.372273    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:19.372368 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.372352    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.372420 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.372404    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.372458 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.372443    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:19.373037 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.373020    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:19.373133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.373049    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:19.373133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.373060    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:19.373133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.373022    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:19.373133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.373127    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:19.373291 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.373141    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:19.374173 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.374159    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.374229 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.374184    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:19.374882 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.374868    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:19.374976 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.374897    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:19.374976 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.374910    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:19.385465 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.385446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1dddbeb549db7d5e9c6f115eb82341f0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-73.ec2.internal\" (UID: \"1dddbeb549db7d5e9c6f115eb82341f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.385554 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.385471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01d8fa5b59ffde43307c653fa5908846-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal\" (UID: \"01d8fa5b59ffde43307c653fa5908846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.385554 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.385487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01d8fa5b59ffde43307c653fa5908846-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal\" (UID: \"01d8fa5b59ffde43307c653fa5908846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.406790 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.406761    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-73.ec2.internal\" not found" node="ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.411139 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.411123    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-73.ec2.internal\" not found" node="ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.472599 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.472571    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.486312 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.486285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1dddbeb549db7d5e9c6f115eb82341f0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-73.ec2.internal\" (UID: \"1dddbeb549db7d5e9c6f115eb82341f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.486370 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.486315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01d8fa5b59ffde43307c653fa5908846-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal\" (UID: \"01d8fa5b59ffde43307c653fa5908846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.486370 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.486347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01d8fa5b59ffde43307c653fa5908846-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal\" (UID: \"01d8fa5b59ffde43307c653fa5908846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.486434 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.486391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01d8fa5b59ffde43307c653fa5908846-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal\" (UID: \"01d8fa5b59ffde43307c653fa5908846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.486434 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.486395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1dddbeb549db7d5e9c6f115eb82341f0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-73.ec2.internal\" (UID: \"1dddbeb549db7d5e9c6f115eb82341f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.486434 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.486396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01d8fa5b59ffde43307c653fa5908846-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal\" (UID: \"01d8fa5b59ffde43307c653fa5908846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.572833 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.572769    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.673327 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.673303    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.708786 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.708752    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.713747 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.713731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.773567 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.773525    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.874185 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:19.874111    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-73.ec2.internal\" not found"
Jul 10 18:40:19.944602 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.944575    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:19.981854 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.981829    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:19.991333 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.991312    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:40:19.993082 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:19.993069    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"
Jul 10 18:40:20.003726 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.003708    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:40:20.013071 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.013045    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 18:40:20.013209 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.013189    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:40:20.013257 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.013228    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:40:20.013257 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.013230    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:40:20.066392 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.066360    2565 apiserver.go:52] "Watching apiserver"
Jul 10 18:40:20.081038 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.081006    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 18:40:20.082747 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.082720    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:35:19 +0000 UTC" deadline="2028-01-02 03:05:53.337779713 +0000 UTC"
Jul 10 18:40:20.082824 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.082747    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12968h25m33.255035204s"
Jul 10 18:40:20.082824 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.082734    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 18:40:20.083813 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.083788    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-6x46v","openshift-network-diagnostics/network-check-target-vfbvz","openshift-network-operator/iptables-alerter-m7t2d","kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669","openshift-image-registry/node-ca-qp9st","openshift-multus/multus-additional-cni-plugins-8lrrk","openshift-multus/network-metrics-daemon-vhp8k","openshift-ovn-kubernetes/ovnkube-node-l79f7","kube-system/global-pull-secret-syncer-2m65d","kube-system/konnectivity-agent-2cjfx","openshift-cluster-node-tuning-operator/tuned-t629v","openshift-dns/node-resolver-wjcdv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal"]
Jul 10 18:40:20.085620 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.085602    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.085699 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.085666    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:20.086737 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.086711    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.087818 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.087797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.088964 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.088941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:20.089078 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.089031    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:20.090060 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.090040    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.090145 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.090056    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 18:40:20.090211 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.090041    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 18:40:20.091219 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.091203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.091637 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.091619    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:20.091725 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.091626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s8ghq\""
Jul 10 18:40:20.091984 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.091966    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:20.092547 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.092534    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.093832 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.093816    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 18:40:20.093921 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.093816    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xbtkr\""
Jul 10 18:40:20.094166 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.094149    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.097929 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.096176    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 18:40:20.097929 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.096300    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:40:20.097929 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.096633    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xjlrg\""
Jul 10 18:40:20.097929 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.096736    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:20.097929 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.096888    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 18:40:20.097929 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.096948    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:20.097929 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.097051    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 18:40:20.097929 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.097810    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 18:40:20.098402 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.098385    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 18:40:20.098522 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.098506    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 18:40:20.099010 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.098989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.100371 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.100354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:20.101443 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.101423    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.106302 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.106180    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6nnz4\""
Jul 10 18:40:20.106302 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.106258    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 18:40:20.106446 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.106314    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 18:40:20.106558 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.106473    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 18:40:20.106558 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.106555    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 18:40:20.106722 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.106575    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 18:40:20.106722 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.106702    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 18:40:20.106788 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.106726    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 18:40:20.107881 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107586    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 18:40:20.107881 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107623    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 18:40:20.107881 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107671    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 18:40:20.107881 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107674    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hv9sp\""
Jul 10 18:40:20.107881 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107586    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 18:40:20.107881 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107863    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 18:40:20.108210 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107863    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z292f\""
Jul 10 18:40:20.108210 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107908    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 18:40:20.108210 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107982    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:20.108210 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.107960    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:20.108210 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.108045    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qkhjt\""
Jul 10 18:40:20.108210 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.108134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 18:40:20.108480 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.108333    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-znqk9\""
Jul 10 18:40:20.108480 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.108374    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ntzfd\""
Jul 10 18:40:20.157623 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.157600    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wlbbd"
Jul 10 18:40:20.172025 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.171998    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wlbbd"
Jul 10 18:40:20.183575 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.183548    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 18:40:20.190134 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:20.190134 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-socket-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.190317 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn9gd\" (UniqueName: \"kubernetes.io/projected/225f1177-70cc-4498-96d5-5fb4b00aa1cb-kube-api-access-mn9gd\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.190317 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-cnibin\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.190317 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzs7\" (UniqueName: \"kubernetes.io/projected/28fbcd0b-2152-4351-9455-60a660f1e83a-kube-api-access-gmzs7\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.190317 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-lib-modules\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.190317 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-host\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2e93258-74e9-4eb4-b466-b3a5141554a2-serviceca\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf5cc605-07e4-464f-a084-1c2f49ae6ebe-agent-certs\") pod \"konnectivity-agent-2cjfx\" (UID: \"cf5cc605-07e4-464f-a084-1c2f49ae6ebe\") " pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-sys-fs\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-systemd\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-etc-kubernetes\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvsgt\" (UniqueName: \"kubernetes.io/projected/e2e93258-74e9-4eb4-b466-b3a5141554a2-kube-api-access-cvsgt\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4qct\" (UniqueName: \"kubernetes.io/projected/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-kube-api-access-p4qct\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:20.190508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vsf\" (UniqueName: \"kubernetes.io/projected/f1100f38-e07c-440d-b610-7f2dc8035c9c-kube-api-access-q5vsf\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-k8s-cni-cncf-io\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a4647ef-1fd0-4d22-9037-298a644990a6-host-slash\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-etc-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-cnibin\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190627    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-tmp-dir\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190642    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-device-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-modprobe-d\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfvt\" (UniqueName: \"kubernetes.io/projected/be89d687-cb34-467b-b5c3-73206230e2ab-kube-api-access-klfvt\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-kubelet-config\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-dbus\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.190811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-system-cni-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-os-release\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9wp\" (UniqueName: \"kubernetes.io/projected/a206ccd3-28b2-41c5-acb7-833d43e07bd3-kube-api-access-xd9wp\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-system-cni-dir\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-os-release\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqp88\" (UniqueName: \"kubernetes.io/projected/4cd99848-3cfb-480a-8c22-b155ccb73066-kube-api-access-hqp88\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a4647ef-1fd0-4d22-9037-298a644990a6-iptables-alerter-script\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysctl-conf\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.190993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-slash\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-node-log\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-cni-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysconfig\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191063    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovn-node-metrics-cert\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf5cc605-07e4-464f-a084-1c2f49ae6ebe-konnectivity-ca\") pod \"konnectivity-agent-2cjfx\" (UID: \"cf5cc605-07e4-464f-a084-1c2f49ae6ebe\") " pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-registration-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.191268 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-ovn\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-netns\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-run\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be89d687-cb34-467b-b5c3-73206230e2ab-etc-tuned\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191241    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be89d687-cb34-467b-b5c3-73206230e2ab-tmp\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-systemd\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191285    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-var-lib-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-socket-dir-parent\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-cni-multus\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-multus-certs\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-kubernetes\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2e93258-74e9-4eb4-b466-b3a5141554a2-host\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-cni-netd\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191587    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-daemon-config\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.192007 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-kubelet\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-env-overrides\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-hosts-file\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-kubelet\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-var-lib-kubelet\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5n92\" (UniqueName: \"kubernetes.io/projected/6a4647ef-1fd0-4d22-9037-298a644990a6-kube-api-access-s5n92\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-systemd-units\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-cni-bin\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysctl-d\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-sys\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-cni-bin\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-conf-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-log-socket\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.191995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.192042    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.192624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.192071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovnkube-config\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.193161 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.192095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28fbcd0b-2152-4351-9455-60a660f1e83a-cni-binary-copy\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.193161 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.192120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-hostroot\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.193161 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.192145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-run-netns\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.193161 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.192170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovnkube-script-lib\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.193161 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.192197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-cni-binary-copy\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.292888 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.292850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-cni-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.292888 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.292890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysconfig\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.292909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovn-node-metrics-cert\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.292927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.292949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf5cc605-07e4-464f-a084-1c2f49ae6ebe-konnectivity-ca\") pod \"konnectivity-agent-2cjfx\" (UID: \"cf5cc605-07e4-464f-a084-1c2f49ae6ebe\") " pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.292990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-registration-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-ovn\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysconfig\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-netns\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-ovn\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-run\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-registration-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-cni-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be89d687-cb34-467b-b5c3-73206230e2ab-etc-tuned\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-run\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be89d687-cb34-467b-b5c3-73206230e2ab-tmp\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.293157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-systemd\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-var-lib-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-socket-dir-parent\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-cni-multus\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293263    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-multus-certs\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-kubernetes\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2e93258-74e9-4eb4-b466-b3a5141554a2-host\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-cni-netd\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293392    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-cni-multus\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293419    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-daemon-config\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-kubelet\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-var-lib-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-netns\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.293801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-multus-certs\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-socket-dir-parent\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf5cc605-07e4-464f-a084-1c2f49ae6ebe-konnectivity-ca\") pod \"konnectivity-agent-2cjfx\" (UID: \"cf5cc605-07e4-464f-a084-1c2f49ae6ebe\") " pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2e93258-74e9-4eb4-b466-b3a5141554a2-host\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-cni-netd\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-kubernetes\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-kubelet\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-run-systemd\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-env-overrides\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-hosts-file\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-kubelet\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.293980    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-env-overrides\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.294902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-var-lib-kubelet\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5n92\" (UniqueName: \"kubernetes.io/projected/6a4647ef-1fd0-4d22-9037-298a644990a6-kube-api-access-s5n92\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-hosts-file\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-kubelet\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-systemd-units\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-var-lib-kubelet\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-cni-bin\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysctl-d\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-systemd-units\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-cni-bin\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-sys\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-cni-bin\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-conf-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysctl-d\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-log-socket\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-var-lib-cni-bin\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-conf-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.295822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-log-socket\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovnkube-config\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28fbcd0b-2152-4351-9455-60a660f1e83a-cni-binary-copy\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-hostroot\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-sys\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-run-netns\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294635    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-hostroot\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovnkube-script-lib\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-cni-binary-copy\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-run-netns\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-socket-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9gd\" (UniqueName: \"kubernetes.io/projected/225f1177-70cc-4498-96d5-5fb4b00aa1cb-kube-api-access-mn9gd\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-socket-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.296765 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-cnibin\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzs7\" (UniqueName: \"kubernetes.io/projected/28fbcd0b-2152-4351-9455-60a660f1e83a-kube-api-access-gmzs7\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.294962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-lib-modules\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-host\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2e93258-74e9-4eb4-b466-b3a5141554a2-serviceca\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf5cc605-07e4-464f-a084-1c2f49ae6ebe-agent-certs\") pod \"konnectivity-agent-2cjfx\" (UID: \"cf5cc605-07e4-464f-a084-1c2f49ae6ebe\") " pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-sys-fs\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-systemd\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28fbcd0b-2152-4351-9455-60a660f1e83a-cni-binary-copy\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-etc-kubernetes\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvsgt\" (UniqueName: \"kubernetes.io/projected/e2e93258-74e9-4eb4-b466-b3a5141554a2-kube-api-access-cvsgt\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-systemd\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovnkube-config\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295307    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-host\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4qct\" (UniqueName: \"kubernetes.io/projected/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-kube-api-access-p4qct\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295351    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-cnibin\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vsf\" (UniqueName: \"kubernetes.io/projected/f1100f38-e07c-440d-b610-7f2dc8035c9c-kube-api-access-q5vsf\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:20.297475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovnkube-script-lib\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-cni-binary-copy\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-sys-fs\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295632    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-etc-kubernetes\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-lib-modules\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-k8s-cni-cncf-io\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a4647ef-1fd0-4d22-9037-298a644990a6-host-slash\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.295710    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295751    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a4647ef-1fd0-4d22-9037-298a644990a6-host-slash\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-etc-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-cnibin\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28fbcd0b-2152-4351-9455-60a660f1e83a-multus-daemon-config\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-tmp-dir\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-cnibin\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-device-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2e93258-74e9-4eb4-b466-b3a5141554a2-serviceca\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.298216 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-modprobe-d\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.295958    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs podName:f1100f38-e07c-440d-b610-7f2dc8035c9c nodeName:}" failed. No retries permitted until 2026-07-10 18:40:20.795920806 +0000 UTC m=+2.094262876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs") pod "network-metrics-daemon-vhp8k" (UID: "f1100f38-e07c-440d-b610-7f2dc8035c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-etc-openvswitch\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.295992    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/225f1177-70cc-4498-96d5-5fb4b00aa1cb-device-dir\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-modprobe-d\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfvt\" (UniqueName: \"kubernetes.io/projected/be89d687-cb34-467b-b5c3-73206230e2ab-kube-api-access-klfvt\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296105    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-host-run-k8s-cni-cncf-io\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-kubelet-config\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-dbus\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-system-cni-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296314    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-os-release\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296331    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-dbus\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9wp\" (UniqueName: \"kubernetes.io/projected/a206ccd3-28b2-41c5-acb7-833d43e07bd3-kube-api-access-xd9wp\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-tmp-dir\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-system-cni-dir\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.298734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd99848-3cfb-480a-8c22-b155ccb73066-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-os-release\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-os-release\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296479    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-kubelet-config\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.296498    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.296547    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret podName:d0e8e075-a99c-4edf-b1ba-3a5504c6734d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:20.796529594 +0000 UTC m=+2.094871667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret") pod "global-pull-secret-syncer-2m65d" (UID: "d0e8e075-a99c-4edf-b1ba-3a5504c6734d") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28fbcd0b-2152-4351-9455-60a660f1e83a-system-cni-dir\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-system-cni-dir\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4cd99848-3cfb-480a-8c22-b155ccb73066-os-release\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqp88\" (UniqueName: \"kubernetes.io/projected/4cd99848-3cfb-480a-8c22-b155ccb73066-kube-api-access-hqp88\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a4647ef-1fd0-4d22-9037-298a644990a6-iptables-alerter-script\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysctl-conf\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-slash\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296886    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be89d687-cb34-467b-b5c3-73206230e2ab-etc-sysctl-conf\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.296927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-node-log\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.297099    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-node-log\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.297123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a206ccd3-28b2-41c5-acb7-833d43e07bd3-host-slash\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.299264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.297458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a4647ef-1fd0-4d22-9037-298a644990a6-iptables-alerter-script\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.299931 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.297460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be89d687-cb34-467b-b5c3-73206230e2ab-tmp\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.299931 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.297628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be89d687-cb34-467b-b5c3-73206230e2ab-etc-tuned\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.299931 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.298485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a206ccd3-28b2-41c5-acb7-833d43e07bd3-ovn-node-metrics-cert\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.299931 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.299016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf5cc605-07e4-464f-a084-1c2f49ae6ebe-agent-certs\") pod \"konnectivity-agent-2cjfx\" (UID: \"cf5cc605-07e4-464f-a084-1c2f49ae6ebe\") " pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:20.313979 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.313948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5n92\" (UniqueName: \"kubernetes.io/projected/6a4647ef-1fd0-4d22-9037-298a644990a6-kube-api-access-s5n92\") pod \"iptables-alerter-m7t2d\" (UID: \"6a4647ef-1fd0-4d22-9037-298a644990a6\") " pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.320304 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.320282    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:20.320463 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.320310    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:20.320463 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.320323    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ctzsk for pod openshift-network-diagnostics/network-check-target-vfbvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:20.320463 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.320377    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk podName:3be1ba28-5b3d-4f64-9575-5e98a563732d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:20.820361465 +0000 UTC m=+2.118703513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ctzsk" (UniqueName: "kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk") pod "network-check-target-vfbvz" (UID: "3be1ba28-5b3d-4f64-9575-5e98a563732d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:20.322325 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.322305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvsgt\" (UniqueName: \"kubernetes.io/projected/e2e93258-74e9-4eb4-b466-b3a5141554a2-kube-api-access-cvsgt\") pod \"node-ca-qp9st\" (UID: \"e2e93258-74e9-4eb4-b466-b3a5141554a2\") " pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.324085 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.324065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9gd\" (UniqueName: \"kubernetes.io/projected/225f1177-70cc-4498-96d5-5fb4b00aa1cb-kube-api-access-mn9gd\") pod \"aws-ebs-csi-driver-node-xp669\" (UID: \"225f1177-70cc-4498-96d5-5fb4b00aa1cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.339892 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.339867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzs7\" (UniqueName: \"kubernetes.io/projected/28fbcd0b-2152-4351-9455-60a660f1e83a-kube-api-access-gmzs7\") pod \"multus-6x46v\" (UID: \"28fbcd0b-2152-4351-9455-60a660f1e83a\") " pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.344389 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.344358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4qct\" (UniqueName: \"kubernetes.io/projected/fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6-kube-api-access-p4qct\") pod \"node-resolver-wjcdv\" (UID: \"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6\") " pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.344935 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.344850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9wp\" (UniqueName: \"kubernetes.io/projected/a206ccd3-28b2-41c5-acb7-833d43e07bd3-kube-api-access-xd9wp\") pod \"ovnkube-node-l79f7\" (UID: \"a206ccd3-28b2-41c5-acb7-833d43e07bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.346600 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.346575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqp88\" (UniqueName: \"kubernetes.io/projected/4cd99848-3cfb-480a-8c22-b155ccb73066-kube-api-access-hqp88\") pod \"multus-additional-cni-plugins-8lrrk\" (UID: \"4cd99848-3cfb-480a-8c22-b155ccb73066\") " pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.346717 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.346698    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vsf\" (UniqueName: \"kubernetes.io/projected/f1100f38-e07c-440d-b610-7f2dc8035c9c-kube-api-access-q5vsf\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:20.348464 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.348441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfvt\" (UniqueName: \"kubernetes.io/projected/be89d687-cb34-467b-b5c3-73206230e2ab-kube-api-access-klfvt\") pod \"tuned-t629v\" (UID: \"be89d687-cb34-467b-b5c3-73206230e2ab\") " pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.350032 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.349988    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dddbeb549db7d5e9c6f115eb82341f0.slice/crio-517ef3fa1053a90de263b01df73422ac0e4fb5a62a330915266c9f894b593f5d WatchSource:0}: Error finding container 517ef3fa1053a90de263b01df73422ac0e4fb5a62a330915266c9f894b593f5d: Status 404 returned error can't find the container with id 517ef3fa1053a90de263b01df73422ac0e4fb5a62a330915266c9f894b593f5d
Jul 10 18:40:20.350620 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.350593    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d8fa5b59ffde43307c653fa5908846.slice/crio-38b11c8cc70c7d41752d8afc93ddc9ed8b4437d737dcff3bb7ceac9183caaead WatchSource:0}: Error finding container 38b11c8cc70c7d41752d8afc93ddc9ed8b4437d737dcff3bb7ceac9183caaead: Status 404 returned error can't find the container with id 38b11c8cc70c7d41752d8afc93ddc9ed8b4437d737dcff3bb7ceac9183caaead
Jul 10 18:40:20.355078 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.355063    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:40:20.406891 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.406804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wjcdv"
Jul 10 18:40:20.415775 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.415746    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2831cc_2294_4c4e_9cb4_cee0fbc3a2f6.slice/crio-f656b83ed8398e775dbf314e77ea4e5f6a4c9d543e3e1b31ac3aec55df3f0f1a WatchSource:0}: Error finding container f656b83ed8398e775dbf314e77ea4e5f6a4c9d543e3e1b31ac3aec55df3f0f1a: Status 404 returned error can't find the container with id f656b83ed8398e775dbf314e77ea4e5f6a4c9d543e3e1b31ac3aec55df3f0f1a
Jul 10 18:40:20.421195 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.421174    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m7t2d"
Jul 10 18:40:20.426592 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.426568    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:20.428753 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.428727    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4647ef_1fd0_4d22_9037_298a644990a6.slice/crio-3efbb11af8bbd022ac4dd371edb516fc1c1dbcf084acb07d0b0fd4b5b4ebd11b WatchSource:0}: Error finding container 3efbb11af8bbd022ac4dd371edb516fc1c1dbcf084acb07d0b0fd4b5b4ebd11b: Status 404 returned error can't find the container with id 3efbb11af8bbd022ac4dd371edb516fc1c1dbcf084acb07d0b0fd4b5b4ebd11b
Jul 10 18:40:20.435650 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.435616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669"
Jul 10 18:40:20.443763 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.443738    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225f1177_70cc_4498_96d5_5fb4b00aa1cb.slice/crio-732ab91730fdf12f3a9f4873db7c9780564b09dd5f537e0e1691c5a143f90f9e WatchSource:0}: Error finding container 732ab91730fdf12f3a9f4873db7c9780564b09dd5f537e0e1691c5a143f90f9e: Status 404 returned error can't find the container with id 732ab91730fdf12f3a9f4873db7c9780564b09dd5f537e0e1691c5a143f90f9e
Jul 10 18:40:20.455667 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.455643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qp9st"
Jul 10 18:40:20.464858 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.464820    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e93258_74e9_4eb4_b466_b3a5141554a2.slice/crio-2a36c67a16b4a5e63295a6165c6528d5253c037e617e003f5cfe8fa16a9153bc WatchSource:0}: Error finding container 2a36c67a16b4a5e63295a6165c6528d5253c037e617e003f5cfe8fa16a9153bc: Status 404 returned error can't find the container with id 2a36c67a16b4a5e63295a6165c6528d5253c037e617e003f5cfe8fa16a9153bc
Jul 10 18:40:20.476593 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.476566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:20.485066 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.485032    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda206ccd3_28b2_41c5_acb7_833d43e07bd3.slice/crio-db40aa0b0436462d0ff54330349bfd4be3ec76d5c3ad3203843630b72097c8de WatchSource:0}: Error finding container db40aa0b0436462d0ff54330349bfd4be3ec76d5c3ad3203843630b72097c8de: Status 404 returned error can't find the container with id db40aa0b0436462d0ff54330349bfd4be3ec76d5c3ad3203843630b72097c8de
Jul 10 18:40:20.498482 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.498461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8lrrk"
Jul 10 18:40:20.506630 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.506607    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd99848_3cfb_480a_8c22_b155ccb73066.slice/crio-850e222de9f63e3a0c59188b55c63e473a775f02a2fb5a51f9cb4dc3a27bcca3 WatchSource:0}: Error finding container 850e222de9f63e3a0c59188b55c63e473a775f02a2fb5a51f9cb4dc3a27bcca3: Status 404 returned error can't find the container with id 850e222de9f63e3a0c59188b55c63e473a775f02a2fb5a51f9cb4dc3a27bcca3
Jul 10 18:40:20.514258 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.514234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6x46v"
Jul 10 18:40:20.522244 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.522218    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fbcd0b_2152_4351_9455_60a660f1e83a.slice/crio-82e0365bf6dbe6c76df432280608a70889694ba271818195711445e14d8cdc3e WatchSource:0}: Error finding container 82e0365bf6dbe6c76df432280608a70889694ba271818195711445e14d8cdc3e: Status 404 returned error can't find the container with id 82e0365bf6dbe6c76df432280608a70889694ba271818195711445e14d8cdc3e
Jul 10 18:40:20.538095 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.538071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:20.544859 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.544834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t629v"
Jul 10 18:40:20.548787 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.548762    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5cc605_07e4_464f_a084_1c2f49ae6ebe.slice/crio-9c4d0fa7e242bb119508be3ac4c40ab8ae4a8fd94cbb426d4f308718e29a6222 WatchSource:0}: Error finding container 9c4d0fa7e242bb119508be3ac4c40ab8ae4a8fd94cbb426d4f308718e29a6222: Status 404 returned error can't find the container with id 9c4d0fa7e242bb119508be3ac4c40ab8ae4a8fd94cbb426d4f308718e29a6222
Jul 10 18:40:20.553147 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:40:20.553127    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe89d687_cb34_467b_b5c3_73206230e2ab.slice/crio-d3adf40de6d6a08e07adaacc961cc17cc98e07946f9cc5a3c1bc331f19ff026b WatchSource:0}: Error finding container d3adf40de6d6a08e07adaacc961cc17cc98e07946f9cc5a3c1bc331f19ff026b: Status 404 returned error can't find the container with id d3adf40de6d6a08e07adaacc961cc17cc98e07946f9cc5a3c1bc331f19ff026b
Jul 10 18:40:20.554782 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.554762    2565 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-t629v_openshift-cluster-node-tuning-operator(be89d687-cb34-467b-b5c3-73206230e2ab): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 10 18:40:20.554833 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.554797    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-t629v" podUID="be89d687-cb34-467b-b5c3-73206230e2ab"
Jul 10 18:40:20.800577 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.800490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:20.800577 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.800532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:20.800760 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.800666    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:20.800760 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.800666    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:20.800760 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.800747    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret podName:d0e8e075-a99c-4edf-b1ba-3a5504c6734d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:21.800726015 +0000 UTC m=+3.099068086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret") pod "global-pull-secret-syncer-2m65d" (UID: "d0e8e075-a99c-4edf-b1ba-3a5504c6734d") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:20.800861 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.800763    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs podName:f1100f38-e07c-440d-b610-7f2dc8035c9c nodeName:}" failed. No retries permitted until 2026-07-10 18:40:21.800756322 +0000 UTC m=+3.099098370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs") pod "network-metrics-daemon-vhp8k" (UID: "f1100f38-e07c-440d-b610-7f2dc8035c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:20.900963 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:20.900930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:20.901169 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.901119    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:20.901169 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.901141    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:20.901169 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.901154    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ctzsk for pod openshift-network-diagnostics/network-check-target-vfbvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:20.901330 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:20.901227    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk podName:3be1ba28-5b3d-4f64-9575-5e98a563732d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:21.901195969 +0000 UTC m=+3.199538040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ctzsk" (UniqueName: "kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk") pod "network-check-target-vfbvz" (UID: "3be1ba28-5b3d-4f64-9575-5e98a563732d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:21.174439 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.174312    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:35:20 +0000 UTC" deadline="2028-02-06 04:21:06.99917414 +0000 UTC"
Jul 10 18:40:21.174439 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.174377    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13809h40m45.824802146s"
Jul 10 18:40:21.180874 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.180843    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:21.270173 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.270128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:21.270343 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.270284    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:21.270923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.270902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:21.271052 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.271032    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:21.276008 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.275889    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" event={"ID":"4cd99848-3cfb-480a-8c22-b155ccb73066","Type":"ContainerStarted","Data":"850e222de9f63e3a0c59188b55c63e473a775f02a2fb5a51f9cb4dc3a27bcca3"}
Jul 10 18:40:21.279241 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.279163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qp9st" event={"ID":"e2e93258-74e9-4eb4-b466-b3a5141554a2","Type":"ContainerStarted","Data":"2a36c67a16b4a5e63295a6165c6528d5253c037e617e003f5cfe8fa16a9153bc"}
Jul 10 18:40:21.282343 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.282278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669" event={"ID":"225f1177-70cc-4498-96d5-5fb4b00aa1cb","Type":"ContainerStarted","Data":"732ab91730fdf12f3a9f4873db7c9780564b09dd5f537e0e1691c5a143f90f9e"}
Jul 10 18:40:21.287925 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.287896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m7t2d" event={"ID":"6a4647ef-1fd0-4d22-9037-298a644990a6","Type":"ContainerStarted","Data":"3efbb11af8bbd022ac4dd371edb516fc1c1dbcf084acb07d0b0fd4b5b4ebd11b"}
Jul 10 18:40:21.289719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.289691    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wjcdv" event={"ID":"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6","Type":"ContainerStarted","Data":"f656b83ed8398e775dbf314e77ea4e5f6a4c9d543e3e1b31ac3aec55df3f0f1a"}
Jul 10 18:40:21.297361 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.297326    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"db40aa0b0436462d0ff54330349bfd4be3ec76d5c3ad3203843630b72097c8de"}
Jul 10 18:40:21.300903 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.300836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal" event={"ID":"01d8fa5b59ffde43307c653fa5908846","Type":"ContainerStarted","Data":"38b11c8cc70c7d41752d8afc93ddc9ed8b4437d737dcff3bb7ceac9183caaead"}
Jul 10 18:40:21.307468 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.307306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal" event={"ID":"1dddbeb549db7d5e9c6f115eb82341f0","Type":"ContainerStarted","Data":"517ef3fa1053a90de263b01df73422ac0e4fb5a62a330915266c9f894b593f5d"}
Jul 10 18:40:21.311400 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.310689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t629v" event={"ID":"be89d687-cb34-467b-b5c3-73206230e2ab","Type":"ContainerStarted","Data":"d3adf40de6d6a08e07adaacc961cc17cc98e07946f9cc5a3c1bc331f19ff026b"}
Jul 10 18:40:21.312343 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.312313    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-t629v" podUID="be89d687-cb34-467b-b5c3-73206230e2ab"
Jul 10 18:40:21.313269 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.313241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2cjfx" event={"ID":"cf5cc605-07e4-464f-a084-1c2f49ae6ebe","Type":"ContainerStarted","Data":"9c4d0fa7e242bb119508be3ac4c40ab8ae4a8fd94cbb426d4f308718e29a6222"}
Jul 10 18:40:21.315728 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.315705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6x46v" event={"ID":"28fbcd0b-2152-4351-9455-60a660f1e83a","Type":"ContainerStarted","Data":"82e0365bf6dbe6c76df432280608a70889694ba271818195711445e14d8cdc3e"}
Jul 10 18:40:21.417326 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.417291    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:21.665333 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.665231    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:21.809214 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.808953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:21.809214 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.809033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:21.809214 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.809176    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:21.809526 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.809245    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret podName:d0e8e075-a99c-4edf-b1ba-3a5504c6734d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:23.809225371 +0000 UTC m=+5.107567425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret") pod "global-pull-secret-syncer-2m65d" (UID: "d0e8e075-a99c-4edf-b1ba-3a5504c6734d") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:21.809895 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.809664    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:21.809895 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.809723    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs podName:f1100f38-e07c-440d-b610-7f2dc8035c9c nodeName:}" failed. No retries permitted until 2026-07-10 18:40:23.809704895 +0000 UTC m=+5.108046970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs") pod "network-metrics-daemon-vhp8k" (UID: "f1100f38-e07c-440d-b610-7f2dc8035c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:21.909872 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:21.909826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:21.910103 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.910052    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:21.910103 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.910081    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:21.910103 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.910098    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ctzsk for pod openshift-network-diagnostics/network-check-target-vfbvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:21.910281 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:21.910171    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk podName:3be1ba28-5b3d-4f64-9575-5e98a563732d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:23.910148747 +0000 UTC m=+5.208490801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ctzsk" (UniqueName: "kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk") pod "network-check-target-vfbvz" (UID: "3be1ba28-5b3d-4f64-9575-5e98a563732d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:22.175223 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:22.175177    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:35:20 +0000 UTC" deadline="2027-12-12 17:03:07.368606004 +0000 UTC"
Jul 10 18:40:22.175223 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:22.175219    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12478h22m45.193391928s"
Jul 10 18:40:22.268707 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:22.268197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:22.268707 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:22.268345    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:23.268745 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:23.268484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:23.269253 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.268852    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:23.269325 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:23.269280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:23.269379 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.269362    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:23.839219 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:23.838421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:23.839219 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:23.838478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:23.839219 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.838638    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:23.839219 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.838702    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret podName:d0e8e075-a99c-4edf-b1ba-3a5504c6734d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:27.838685021 +0000 UTC m=+9.137027070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret") pod "global-pull-secret-syncer-2m65d" (UID: "d0e8e075-a99c-4edf-b1ba-3a5504c6734d") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:23.839219 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.839121    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:23.839219 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.839174    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs podName:f1100f38-e07c-440d-b610-7f2dc8035c9c nodeName:}" failed. No retries permitted until 2026-07-10 18:40:27.839158553 +0000 UTC m=+9.137500606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs") pod "network-metrics-daemon-vhp8k" (UID: "f1100f38-e07c-440d-b610-7f2dc8035c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:23.939571 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:23.939082    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:23.939571 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.939235    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:23.939571 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.939249    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:23.939571 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.939258    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ctzsk for pod openshift-network-diagnostics/network-check-target-vfbvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:23.939571 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:23.939302    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk podName:3be1ba28-5b3d-4f64-9575-5e98a563732d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:27.939289901 +0000 UTC m=+9.237631949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ctzsk" (UniqueName: "kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk") pod "network-check-target-vfbvz" (UID: "3be1ba28-5b3d-4f64-9575-5e98a563732d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:24.268673 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:24.268586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:24.268834 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:24.268724    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:25.268362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:25.268326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:25.268555 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:25.268326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:25.268555 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:25.268470    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:25.268659 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:25.268557    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:26.268315 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:26.268280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:26.268777 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:26.268424    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:27.270388 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:27.269799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:27.270388 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.269906    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:27.270388 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:27.270252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:27.270388 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.270326    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:27.872393 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:27.872347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:27.872586 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:27.872405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:27.872586 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.872533    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:27.872710 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.872598    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret podName:d0e8e075-a99c-4edf-b1ba-3a5504c6734d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:35.872577954 +0000 UTC m=+17.170920003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret") pod "global-pull-secret-syncer-2m65d" (UID: "d0e8e075-a99c-4edf-b1ba-3a5504c6734d") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:27.872930 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.872913    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:27.873027 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.872984    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs podName:f1100f38-e07c-440d-b610-7f2dc8035c9c nodeName:}" failed. No retries permitted until 2026-07-10 18:40:35.872968048 +0000 UTC m=+17.171310112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs") pod "network-metrics-daemon-vhp8k" (UID: "f1100f38-e07c-440d-b610-7f2dc8035c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:27.973386 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:27.973343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:27.973586 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.973558    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:27.973586 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.973577    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:27.973586 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.973587    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ctzsk for pod openshift-network-diagnostics/network-check-target-vfbvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:27.973718 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:27.973645    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk podName:3be1ba28-5b3d-4f64-9575-5e98a563732d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:35.97362633 +0000 UTC m=+17.271968400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ctzsk" (UniqueName: "kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk") pod "network-check-target-vfbvz" (UID: "3be1ba28-5b3d-4f64-9575-5e98a563732d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:28.268344 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:28.268259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:28.268498 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:28.268392    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:29.270679 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:29.270074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:29.270679 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:29.270184    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:29.270679 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:29.270545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:29.270679 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:29.270628    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:30.268673 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:30.268582    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:30.268829 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:30.268729    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:31.268343 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:31.268308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:31.268802 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:31.268446    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:31.268802 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:31.268507    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:31.268802 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:31.268633    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:32.268486 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:32.268450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:32.268914 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:32.268590    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:33.267942 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:33.267905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:33.268141 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:33.267913    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:33.268141 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:33.268031    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:33.268141 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:33.268088    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:34.268445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:34.268405    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:34.268878 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:34.268559    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:35.268013 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:35.267981    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:35.268208 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:35.267996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:35.268208 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:35.268098    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:35.268208 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:35.268163    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:35.934357 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:35.934312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:35.934357 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:35.934352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:35.934826 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:35.934449    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:35.934826 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:35.934470    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:35.934826 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:35.934512    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret podName:d0e8e075-a99c-4edf-b1ba-3a5504c6734d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:51.934496525 +0000 UTC m=+33.232838574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret") pod "global-pull-secret-syncer-2m65d" (UID: "d0e8e075-a99c-4edf-b1ba-3a5504c6734d") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:35.934826 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:35.934538    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs podName:f1100f38-e07c-440d-b610-7f2dc8035c9c nodeName:}" failed. No retries permitted until 2026-07-10 18:40:51.934520653 +0000 UTC m=+33.232862714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs") pod "network-metrics-daemon-vhp8k" (UID: "f1100f38-e07c-440d-b610-7f2dc8035c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:36.035104 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:36.035073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:36.035284 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:36.035208    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:36.035284 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:36.035228    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:36.035284 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:36.035243    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ctzsk for pod openshift-network-diagnostics/network-check-target-vfbvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:36.035435 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:36.035313    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk podName:3be1ba28-5b3d-4f64-9575-5e98a563732d nodeName:}" failed. No retries permitted until 2026-07-10 18:40:52.035293068 +0000 UTC m=+33.333635138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ctzsk" (UniqueName: "kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk") pod "network-check-target-vfbvz" (UID: "3be1ba28-5b3d-4f64-9575-5e98a563732d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:36.267906 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:36.267819    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:36.268131 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:36.267972    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:37.267822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:37.267777    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:37.268380 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:37.267913    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:37.268380 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:37.267978    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:37.268380 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:37.268070    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:38.268009 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:38.267972    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:38.268462 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:38.268106    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:39.269245 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:39.269156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:39.269671 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:39.269269    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:39.269671 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:39.269322    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:39.269671 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:39.269417    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:40.268129 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:40.268098    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:40.268301 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:40.268215    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:41.268127 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.267683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:41.268919 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.267683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:41.268919 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:41.268214    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:41.268919 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:41.268259    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:41.365858 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.365829    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wjcdv" event={"ID":"fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6","Type":"ContainerStarted","Data":"d4dc417c99e358acfae4369d1fe9f8fd39070968426a8fb564456be7155f49cd"}
Jul 10 18:40:41.368124 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.368105    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:40:41.368435 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.368412    2565 generic.go:356] "Generic (PLEG): container finished" podID="a206ccd3-28b2-41c5-acb7-833d43e07bd3" containerID="990e13ea3ed5f3c34743b2cac35f5a6a61ae8c07d4993267e707e14d715c908e" exitCode=1
Jul 10 18:40:41.368512 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.368474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"958766d5d73c2669930dc5d5a129019cc40170f11d3d30b54ab506a44a9afc47"}
Jul 10 18:40:41.368557 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.368523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"d7e69096cc89e6785883c95e14ea9dfada834b61db27c4d8fd3b389b10fd9737"}
Jul 10 18:40:41.368557 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.368554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerDied","Data":"990e13ea3ed5f3c34743b2cac35f5a6a61ae8c07d4993267e707e14d715c908e"}
Jul 10 18:40:41.368649 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.368590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"6873cb60cec76593c4868a05494f2ee3367ee6a8e7883b4ceb0b4b6bb3bf9510"}
Jul 10 18:40:41.369969 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.369928    2565 generic.go:356] "Generic (PLEG): container finished" podID="01d8fa5b59ffde43307c653fa5908846" containerID="b4f04b88065cf27ddd197ebb0fd58ad1c41ec361098c2edfefb4da254ce83c08" exitCode=0
Jul 10 18:40:41.370036 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.369993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal" event={"ID":"01d8fa5b59ffde43307c653fa5908846","Type":"ContainerDied","Data":"b4f04b88065cf27ddd197ebb0fd58ad1c41ec361098c2edfefb4da254ce83c08"}
Jul 10 18:40:41.371481 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.371462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal" event={"ID":"1dddbeb549db7d5e9c6f115eb82341f0","Type":"ContainerStarted","Data":"61ccfa4154923be03d0300509e2274474f600b78236429c7601a45b607848418"}
Jul 10 18:40:41.372846 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.372824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t629v" event={"ID":"be89d687-cb34-467b-b5c3-73206230e2ab","Type":"ContainerStarted","Data":"485da8a3c5d72476d04ff06dd9b2e4960b334eb6b62b876b7cf1fe9900908971"}
Jul 10 18:40:41.374095 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.374075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2cjfx" event={"ID":"cf5cc605-07e4-464f-a084-1c2f49ae6ebe","Type":"ContainerStarted","Data":"08d2bc1d20a5cdfedc5544a3d699d4133c26afccbd01cb0210a5da9c53890c1d"}
Jul 10 18:40:41.375336 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.375316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6x46v" event={"ID":"28fbcd0b-2152-4351-9455-60a660f1e83a","Type":"ContainerStarted","Data":"2fd15531da78396ec7ed0d284c8074e049caec1384018ed3e4528771a2c40003"}
Jul 10 18:40:41.376702 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.376682    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cd99848-3cfb-480a-8c22-b155ccb73066" containerID="e0d1f78bc157dd4048418e233df68c761269349fc4c889b3d46a83eb70453599" exitCode=0
Jul 10 18:40:41.376786 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.376763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" event={"ID":"4cd99848-3cfb-480a-8c22-b155ccb73066","Type":"ContainerDied","Data":"e0d1f78bc157dd4048418e233df68c761269349fc4c889b3d46a83eb70453599"}
Jul 10 18:40:41.378226 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.378032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qp9st" event={"ID":"e2e93258-74e9-4eb4-b466-b3a5141554a2","Type":"ContainerStarted","Data":"692675ef03c5642ff7a8e8449dc93fbfa4c3022425d4b40c4e4a8aa727495ec3"}
Jul 10 18:40:41.379322 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.379308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669" event={"ID":"225f1177-70cc-4498-96d5-5fb4b00aa1cb","Type":"ContainerStarted","Data":"78a6b183546c60accce404fbc1d1a35fb0e67ebda965ffec1b0ee852a2280a9e"}
Jul 10 18:40:41.386339 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.386298    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wjcdv" podStartSLOduration=3.975918594 podStartE2EDuration="22.386286242s" totalImagesPullingTime="18.410367648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:41.38612671 +0000 UTC m=+22.684468779" watchObservedRunningTime="2026-07-10 18:40:41.386286242 +0000 UTC m=+22.684628310"
Jul 10 18:40:41.402647 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.402600    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6x46v" podStartSLOduration=4.043516638 podStartE2EDuration="22.40258698s" totalImagesPullingTime="18.359070342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:41.402191473 +0000 UTC m=+22.700533543" watchObservedRunningTime="2026-07-10 18:40:41.40258698 +0000 UTC m=+22.700929056"
Jul 10 18:40:41.417899 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.417843    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2cjfx" podStartSLOduration=4.098775527 podStartE2EDuration="22.4178241s" totalImagesPullingTime="18.319048573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:41.417239897 +0000 UTC m=+22.715581966" watchObservedRunningTime="2026-07-10 18:40:41.4178241 +0000 UTC m=+22.716166171"
Jul 10 18:40:41.432770 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.432735    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t629v" podStartSLOduration=2.112196571 podStartE2EDuration="22.432722135s" totalImagesPullingTime="20.320525564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-10 18:40:41.432219654 +0000 UTC m=+22.730561724" watchObservedRunningTime="2026-07-10 18:40:41.432722135 +0000 UTC m=+22.731064204"
Jul 10 18:40:41.444860 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.444824    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-73.ec2.internal" podStartSLOduration=21.444810665 podStartE2EDuration="21.444810665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:41.444792035 +0000 UTC m=+22.743134104" watchObservedRunningTime="2026-07-10 18:40:41.444810665 +0000 UTC m=+22.743152733"
Jul 10 18:40:41.457934 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.457890    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qp9st" podStartSLOduration=4.096718508 podStartE2EDuration="22.457876886s" totalImagesPullingTime="18.361158378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:41.457811905 +0000 UTC m=+22.756153973" watchObservedRunningTime="2026-07-10 18:40:41.457876886 +0000 UTC m=+22.756218955"
Jul 10 18:40:41.540277 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.540253    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:41.952363 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:41.952332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:42.158903 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.158824    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:42.267979 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.267935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:42.268131 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:42.268100    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:42.384014 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.383973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m7t2d" event={"ID":"6a4647ef-1fd0-4d22-9037-298a644990a6","Type":"ContainerStarted","Data":"66523d7da8dbde77b79382ae1babd40d78ef79ce5beedd4d04983e59a7e2b9cd"}
Jul 10 18:40:42.388059 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.388033    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:40:42.388630 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.388595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"83d37faf7339c328995e114d33ac97fdde67f4a22a5c72843c7ac11a34a60b9a"}
Jul 10 18:40:42.388746 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.388679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"3538610540be8ba4ea1fa2909434a93d6d936b4ec41287a0fa21c2afd1c764ac"}
Jul 10 18:40:42.399069 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.399029    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m7t2d" podStartSLOduration=4.956704603 podStartE2EDuration="23.399015015s" totalImagesPullingTime="18.442310412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:42.398536071 +0000 UTC m=+23.696878141" watchObservedRunningTime="2026-07-10 18:40:42.399015015 +0000 UTC m=+23.697357084"
Jul 10 18:40:42.406863 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.406836    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:42.407333 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.407312    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:42.739969 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.739860    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:42.892358 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:42.892334    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 18:40:43.138019 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.137934    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:43.165879 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.165768    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:40:42.892353356Z","UUID":"32b5ad60-8b23-4f2a-92f9-4fc18c34020b","Handler":null,"Name":"","Endpoint":""}
Jul 10 18:40:43.167294 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.167276    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 18:40:43.167395 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.167311    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 18:40:43.268533 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.268043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:43.268533 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.268074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:43.268533 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:43.268186    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:43.268533 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:43.268315    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:43.392687 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.392599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669" event={"ID":"225f1177-70cc-4498-96d5-5fb4b00aa1cb","Type":"ContainerStarted","Data":"bc5452ff4be223515ef99287cd62a790b03908a550a02b484a4400ffb14b76a0"}
Jul 10 18:40:43.394636 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.394604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal" event={"ID":"01d8fa5b59ffde43307c653fa5908846","Type":"ContainerStarted","Data":"17a157fc5d0e7964f47f26f225581b47e9ad10e00785ac64cee8454d2de51eac"}
Jul 10 18:40:43.395291 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.394894    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:43.395401 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.395356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2cjfx"
Jul 10 18:40:43.425232 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.423699    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-73.ec2.internal" podStartSLOduration=23.42367366 podStartE2EDuration="23.42367366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:43.409276643 +0000 UTC m=+24.707618711" watchObservedRunningTime="2026-07-10 18:40:43.42367366 +0000 UTC m=+24.722015729"
Jul 10 18:40:43.535848 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.535813    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:43.944924 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:43.944888    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:44.141604 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:44.141570    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:44.268009 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:44.267985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:44.268140 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:44.268097    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:44.336123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:44.336090    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:44.399258 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:44.399218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669" event={"ID":"225f1177-70cc-4498-96d5-5fb4b00aa1cb","Type":"ContainerStarted","Data":"7d1944d1b280e2a4a8d8218befa30e9d92c195f31273b0f1f67f246812467c2f"}
Jul 10 18:40:44.403054 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:44.403028    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:40:44.403496 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:44.403465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"3aa9e4278663dc015e4a89d1870e91b7512e4a819e3b327487999dea84ebe865"}
Jul 10 18:40:44.416537 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:44.416488    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp669" podStartSLOduration=4.134189848 podStartE2EDuration="25.416473356s" totalImagesPullingTime="21.282283508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:44.416176091 +0000 UTC m=+25.714518160" watchObservedRunningTime="2026-07-10 18:40:44.416473356 +0000 UTC m=+25.714815425"
Jul 10 18:40:45.158582 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:45.158388    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:45.265188 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:45.265153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:45.268228 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:45.268207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:45.268371 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:45.268207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:45.268371 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:45.268339    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:45.268484 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:45.268426    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:46.268198 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.268043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:46.268753 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:46.268290    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:46.409205 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.409175    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cd99848-3cfb-480a-8c22-b155ccb73066" containerID="85df81cca826f99d3514c37175c80af5e429e9556a1ae83e716ad6f867808b10" exitCode=0
Jul 10 18:40:46.409368 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.409260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" event={"ID":"4cd99848-3cfb-480a-8c22-b155ccb73066","Type":"ContainerDied","Data":"85df81cca826f99d3514c37175c80af5e429e9556a1ae83e716ad6f867808b10"}
Jul 10 18:40:46.412929 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.412905    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:40:46.413335 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.413316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"c887e5f46047e2c8aab42d8387b159acec90fe6d81df45f99c7e72faf5c0be30"}
Jul 10 18:40:46.417315 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.413982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:46.417315 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.414112    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:46.417315 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.414778    2565 scope.go:117] "RemoveContainer" containerID="990e13ea3ed5f3c34743b2cac35f5a6a61ae8c07d4993267e707e14d715c908e"
Jul 10 18:40:46.432255 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.432235    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:46.433541 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.433521    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:46.516090 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:46.515984    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:40:47.267768 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:47.267674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:47.267908 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:47.267809    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:47.267908 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:47.267863    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:47.268023 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:47.267978    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:47.419596 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:47.419571    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:40:47.420028 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:47.419978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" event={"ID":"a206ccd3-28b2-41c5-acb7-833d43e07bd3","Type":"ContainerStarted","Data":"7f5ba531de599bd5b2d34d85afd7320235ea7ee05e1ee16121e788720030126d"}
Jul 10 18:40:47.450557 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:47.450509    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7" podStartSLOduration=8.116988193 podStartE2EDuration="28.450491325s" totalImagesPullingTime="20.333503132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:47.450237485 +0000 UTC m=+28.748579553" watchObservedRunningTime="2026-07-10 18:40:47.450491325 +0000 UTC m=+28.748833396"
Jul 10 18:40:48.268003 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:48.267949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:48.268144 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:48.268077    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:48.424681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:48.424647    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cd99848-3cfb-480a-8c22-b155ccb73066" containerID="2d9ee438b560be02db387c7309d9ec50d8252bdee3dfbad5157d3644f186f76d" exitCode=0
Jul 10 18:40:48.425160 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:48.424739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" event={"ID":"4cd99848-3cfb-480a-8c22-b155ccb73066","Type":"ContainerDied","Data":"2d9ee438b560be02db387c7309d9ec50d8252bdee3dfbad5157d3644f186f76d"}
Jul 10 18:40:49.268781 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:49.268745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:49.268944 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:49.268821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:49.268944 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:49.268838    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:49.268944 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:49.268922    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:49.429035 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:49.428999    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cd99848-3cfb-480a-8c22-b155ccb73066" containerID="6b2c174ff761bafbe532928e93594317d3a79aacb82a39eeba839712bbe3d3f7" exitCode=0
Jul 10 18:40:49.429390 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:49.429072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" event={"ID":"4cd99848-3cfb-480a-8c22-b155ccb73066","Type":"ContainerDied","Data":"6b2c174ff761bafbe532928e93594317d3a79aacb82a39eeba839712bbe3d3f7"}
Jul 10 18:40:50.268132 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:50.268083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:50.268287 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:50.268216    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:51.268638 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:51.268597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:51.268638 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:51.268622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:51.269153 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:51.268738    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:51.269153 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:51.268823    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:51.950844 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:51.950814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:51.951025 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:51.950858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:51.951025 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:51.950994    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:51.951148 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:51.951051    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs podName:f1100f38-e07c-440d-b610-7f2dc8035c9c nodeName:}" failed. No retries permitted until 2026-07-10 18:41:23.951037739 +0000 UTC m=+65.249379786 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs") pod "network-metrics-daemon-vhp8k" (UID: "f1100f38-e07c-440d-b610-7f2dc8035c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:51.951148 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:51.950994    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:51.951148 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:51.951093    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret podName:d0e8e075-a99c-4edf-b1ba-3a5504c6734d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:23.95108335 +0000 UTC m=+65.249425397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret") pod "global-pull-secret-syncer-2m65d" (UID: "d0e8e075-a99c-4edf-b1ba-3a5504c6734d") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:52.051683 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:52.051653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:52.051850 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:52.051828    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:52.051850 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:52.051849    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:52.051936 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:52.051859    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ctzsk for pod openshift-network-diagnostics/network-check-target-vfbvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:52.051936 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:52.051921    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk podName:3be1ba28-5b3d-4f64-9575-5e98a563732d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:24.051896693 +0000 UTC m=+65.350238754 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ctzsk" (UniqueName: "kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk") pod "network-check-target-vfbvz" (UID: "3be1ba28-5b3d-4f64-9575-5e98a563732d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:52.268662 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:52.268585    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:52.269133 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:52.268725    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:53.267894 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:53.267861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:53.268116 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:53.268023    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:53.268116 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:53.268052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:53.268229 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:53.268163    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:54.193295 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:54.193266    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:54.268165 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:54.268125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:54.268370 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:54.268239    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:54.443752 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:54.443712    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cd99848-3cfb-480a-8c22-b155ccb73066" containerID="5eed6e42dfd202fa7bf9ed6f2eee180d89990fde4d6c32f68ee2a016216d1cc7" exitCode=0
Jul 10 18:40:54.443934 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:54.443771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" event={"ID":"4cd99848-3cfb-480a-8c22-b155ccb73066","Type":"ContainerDied","Data":"5eed6e42dfd202fa7bf9ed6f2eee180d89990fde4d6c32f68ee2a016216d1cc7"}
Jul 10 18:40:55.268133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:55.268090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:55.268539 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:55.268098    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:55.268539 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:55.268233    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:55.268539 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:55.268307    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:55.448519 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:55.448487    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cd99848-3cfb-480a-8c22-b155ccb73066" containerID="a0ceae6bdd16f573fc9109a0edda4c15ddc6ba88c0f3683fcb0ed67a46e177f1" exitCode=0
Jul 10 18:40:55.448647 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:55.448544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" event={"ID":"4cd99848-3cfb-480a-8c22-b155ccb73066","Type":"ContainerDied","Data":"a0ceae6bdd16f573fc9109a0edda4c15ddc6ba88c0f3683fcb0ed67a46e177f1"}
Jul 10 18:40:56.069717 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:56.069685    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:56.186545 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:56.186510    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:56.268184 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:56.268150    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:56.268548 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:56.268263    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:56.284456 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:56.284425    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:56.382843 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:56.382751    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:56.456341 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:56.456309    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" event={"ID":"4cd99848-3cfb-480a-8c22-b155ccb73066","Type":"ContainerStarted","Data":"fb4e4b0422e3b2c1d571783f712cfd5b927eda06abc09094533bb0c563d8ba04"}
Jul 10 18:40:56.481840 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:56.481794    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8lrrk" podStartSLOduration=7.405409871 podStartE2EDuration="37.481776303s" totalImagesPullingTime="30.076366432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:56.481334896 +0000 UTC m=+37.779676978" watchObservedRunningTime="2026-07-10 18:40:56.481776303 +0000 UTC m=+37.780118375"
Jul 10 18:40:56.482222 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:56.482201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:57.009677 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.009644    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:57.103082 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.103040    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:57.268383 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.268273    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:57.268821 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:57.268411    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:57.268821 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.268436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:57.268821 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:57.268540    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:57.401538 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.401504    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:57.474015 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.473984    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2m65d"]
Jul 10 18:40:57.474162 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.474108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:57.474257 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:57.474231    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:40:57.476337 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.476313    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vhp8k"]
Jul 10 18:40:57.476459 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.476413    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:57.476543 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:57.476522    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:57.488773 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.488740    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vfbvz"]
Jul 10 18:40:57.488901 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.488846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:57.488988 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:57.488948    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:57.693451 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:57.693424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:58.237606 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:58.237570    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:58.598925 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:58.598838    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:58.699849 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:58.699819    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:59.269315 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:59.269284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:40:59.269479 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:59.269394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:40:59.269479 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:59.269402    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:40:59.269621 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:59.269505    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:40:59.269621 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:40:59.269573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:40:59.269724 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:40:59.269651    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:41:00.008841 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:00.008747    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:00.122828 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:00.122793    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:00.627450 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:00.627419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:00.927682 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:00.927655    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:01.030001 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.029969    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:01.268035 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.267961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:41:01.268035 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.268002    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:41:01.268212 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.267963    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:41:01.268212 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:01.268070    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfbvz" podUID="3be1ba28-5b3d-4f64-9575-5e98a563732d"
Jul 10 18:41:01.268212 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:01.268131    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m65d" podUID="d0e8e075-a99c-4edf-b1ba-3a5504c6734d"
Jul 10 18:41:01.268314 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:01.268211    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhp8k" podUID="f1100f38-e07c-440d-b610-7f2dc8035c9c"
Jul 10 18:41:01.495734 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.495702    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-73.ec2.internal" event="NodeReady"
Jul 10 18:41:01.495913 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.495853    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 18:41:01.602621 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.602542    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5f954"]
Jul 10 18:41:01.623712 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.623678    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nxzsc"]
Jul 10 18:41:01.623858 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.623792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.628755 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.628735    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 18:41:01.629020 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.629002    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 18:41:01.629087 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.629070    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwhdh\""
Jul 10 18:41:01.629483 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.629464    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 18:41:01.630176 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.630159    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 18:41:01.637594 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.637572    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5f954"]
Jul 10 18:41:01.637704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.637618    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nxzsc"]
Jul 10 18:41:01.637704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.637640    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z2pk2"]
Jul 10 18:41:01.637704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.637671    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nxzsc"
Jul 10 18:41:01.640767 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.640750    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k5slx\""
Jul 10 18:41:01.640865 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.640770    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 18:41:01.641081 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.641061    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 18:41:01.641174 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.641109    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 18:41:01.655812 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.655789    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.658635 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.658619    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jc4wc\""
Jul 10 18:41:01.658932 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.658916    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 18:41:01.659024 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.659001    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 18:41:01.663509 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.663479    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z2pk2"]
Jul 10 18:41:01.724856 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.724817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x24m\" (UniqueName: \"kubernetes.io/projected/c02d7e07-698b-4351-9db2-0b570b3b8570-kube-api-access-4x24m\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.724856 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.724852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c02d7e07-698b-4351-9db2-0b570b3b8570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.725089 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.724893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c02d7e07-698b-4351-9db2-0b570b3b8570-data-volume\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.725089 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.724936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c02d7e07-698b-4351-9db2-0b570b3b8570-crio-socket\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.725089 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.725000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c02d7e07-698b-4351-9db2-0b570b3b8570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.739762 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.739734    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:01.825971 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.825916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c02d7e07-698b-4351-9db2-0b570b3b8570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.826123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.825983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlfc\" (UniqueName: \"kubernetes.io/projected/c4f27378-5bf0-4d9a-9508-06c76b584cdc-kube-api-access-8zlfc\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.826123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxfj\" (UniqueName: \"kubernetes.io/projected/fd2847f5-9eae-4319-a9e4-061d9d3f2da1-kube-api-access-2mxfj\") pod \"ingress-canary-nxzsc\" (UID: \"fd2847f5-9eae-4319-a9e4-061d9d3f2da1\") " pod="openshift-ingress-canary/ingress-canary-nxzsc"
Jul 10 18:41:01.826123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f27378-5bf0-4d9a-9508-06c76b584cdc-tmp-dir\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.826123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x24m\" (UniqueName: \"kubernetes.io/projected/c02d7e07-698b-4351-9db2-0b570b3b8570-kube-api-access-4x24m\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.826123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826068    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f27378-5bf0-4d9a-9508-06c76b584cdc-config-volume\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.826123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c02d7e07-698b-4351-9db2-0b570b3b8570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.826123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd2847f5-9eae-4319-a9e4-061d9d3f2da1-cert\") pod \"ingress-canary-nxzsc\" (UID: \"fd2847f5-9eae-4319-a9e4-061d9d3f2da1\") " pod="openshift-ingress-canary/ingress-canary-nxzsc"
Jul 10 18:41:01.826356 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c02d7e07-698b-4351-9db2-0b570b3b8570-data-volume\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.826356 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4f27378-5bf0-4d9a-9508-06c76b584cdc-metrics-tls\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.826356 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c02d7e07-698b-4351-9db2-0b570b3b8570-crio-socket\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.826454 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826418    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c02d7e07-698b-4351-9db2-0b570b3b8570-crio-socket\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.826492 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c02d7e07-698b-4351-9db2-0b570b3b8570-data-volume\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.826725 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.826708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c02d7e07-698b-4351-9db2-0b570b3b8570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.830392 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.830375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c02d7e07-698b-4351-9db2-0b570b3b8570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.837179 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.837153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:01.839453 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.839429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x24m\" (UniqueName: \"kubernetes.io/projected/c02d7e07-698b-4351-9db2-0b570b3b8570-kube-api-access-4x24m\") pod \"insights-runtime-extractor-5f954\" (UID: \"c02d7e07-698b-4351-9db2-0b570b3b8570\") " pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.927048 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.927017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlfc\" (UniqueName: \"kubernetes.io/projected/c4f27378-5bf0-4d9a-9508-06c76b584cdc-kube-api-access-8zlfc\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.927048 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.927054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxfj\" (UniqueName: \"kubernetes.io/projected/fd2847f5-9eae-4319-a9e4-061d9d3f2da1-kube-api-access-2mxfj\") pod \"ingress-canary-nxzsc\" (UID: \"fd2847f5-9eae-4319-a9e4-061d9d3f2da1\") " pod="openshift-ingress-canary/ingress-canary-nxzsc"
Jul 10 18:41:01.927260 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.927076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f27378-5bf0-4d9a-9508-06c76b584cdc-tmp-dir\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.927260 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.927099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f27378-5bf0-4d9a-9508-06c76b584cdc-config-volume\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.927260 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.927125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd2847f5-9eae-4319-a9e4-061d9d3f2da1-cert\") pod \"ingress-canary-nxzsc\" (UID: \"fd2847f5-9eae-4319-a9e4-061d9d3f2da1\") " pod="openshift-ingress-canary/ingress-canary-nxzsc"
Jul 10 18:41:01.927260 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.927169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4f27378-5bf0-4d9a-9508-06c76b584cdc-metrics-tls\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.927489 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.927465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f27378-5bf0-4d9a-9508-06c76b584cdc-tmp-dir\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.927666 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.927647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f27378-5bf0-4d9a-9508-06c76b584cdc-config-volume\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.929714 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.929690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4f27378-5bf0-4d9a-9508-06c76b584cdc-metrics-tls\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.929823 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.929804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd2847f5-9eae-4319-a9e4-061d9d3f2da1-cert\") pod \"ingress-canary-nxzsc\" (UID: \"fd2847f5-9eae-4319-a9e4-061d9d3f2da1\") " pod="openshift-ingress-canary/ingress-canary-nxzsc"
Jul 10 18:41:01.933676 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.933653    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5f954"
Jul 10 18:41:01.936341 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.936312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlfc\" (UniqueName: \"kubernetes.io/projected/c4f27378-5bf0-4d9a-9508-06c76b584cdc-kube-api-access-8zlfc\") pod \"dns-default-z2pk2\" (UID: \"c4f27378-5bf0-4d9a-9508-06c76b584cdc\") " pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:01.936426 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.936340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxfj\" (UniqueName: \"kubernetes.io/projected/fd2847f5-9eae-4319-a9e4-061d9d3f2da1-kube-api-access-2mxfj\") pod \"ingress-canary-nxzsc\" (UID: \"fd2847f5-9eae-4319-a9e4-061d9d3f2da1\") " pod="openshift-ingress-canary/ingress-canary-nxzsc"
Jul 10 18:41:01.945244 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.945225    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nxzsc"
Jul 10 18:41:01.963809 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:01.963787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:02.202443 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:02.202146    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nxzsc"]
Jul 10 18:41:02.206245 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:02.206210    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2847f5_9eae_4319_a9e4_061d9d3f2da1.slice/crio-ac621cedbbadc711d0f213fdd04d2317ecbddbab34622fd01ed3204aeb10e9bc WatchSource:0}: Error finding container ac621cedbbadc711d0f213fdd04d2317ecbddbab34622fd01ed3204aeb10e9bc: Status 404 returned error can't find the container with id ac621cedbbadc711d0f213fdd04d2317ecbddbab34622fd01ed3204aeb10e9bc
Jul 10 18:41:02.206704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:02.206685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5f954"]
Jul 10 18:41:02.207237 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:02.207212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z2pk2"]
Jul 10 18:41:02.213982 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:02.213590    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f27378_5bf0_4d9a_9508_06c76b584cdc.slice/crio-ef1869bb0dd60e564699595e3b9654c09557f4142d7155ec1c4efb9b01126059 WatchSource:0}: Error finding container ef1869bb0dd60e564699595e3b9654c09557f4142d7155ec1c4efb9b01126059: Status 404 returned error can't find the container with id ef1869bb0dd60e564699595e3b9654c09557f4142d7155ec1c4efb9b01126059
Jul 10 18:41:02.214148 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:02.214113    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02d7e07_698b_4351_9db2_0b570b3b8570.slice/crio-3bb0ab93ee4c1457b9f0bee0d9db59250ee1908353d11b6309ce65682cb4f1d8 WatchSource:0}: Error finding container 3bb0ab93ee4c1457b9f0bee0d9db59250ee1908353d11b6309ce65682cb4f1d8: Status 404 returned error can't find the container with id 3bb0ab93ee4c1457b9f0bee0d9db59250ee1908353d11b6309ce65682cb4f1d8
Jul 10 18:41:02.470236 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:02.470141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5f954" event={"ID":"c02d7e07-698b-4351-9db2-0b570b3b8570","Type":"ContainerStarted","Data":"3bb0ab93ee4c1457b9f0bee0d9db59250ee1908353d11b6309ce65682cb4f1d8"}
Jul 10 18:41:02.471144 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:02.471120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2pk2" event={"ID":"c4f27378-5bf0-4d9a-9508-06c76b584cdc","Type":"ContainerStarted","Data":"ef1869bb0dd60e564699595e3b9654c09557f4142d7155ec1c4efb9b01126059"}
Jul 10 18:41:02.472098 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:02.472076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nxzsc" event={"ID":"fd2847f5-9eae-4319-a9e4-061d9d3f2da1","Type":"ContainerStarted","Data":"ac621cedbbadc711d0f213fdd04d2317ecbddbab34622fd01ed3204aeb10e9bc"}
Jul 10 18:41:03.272264 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.272233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:41:03.273128 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.273081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:41:03.273128 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.273108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:41:03.276710 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.276429    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:41:03.276837 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.276813    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:41:03.279531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.279059    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6dtsl\""
Jul 10 18:41:03.279531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.279173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:41:03.279531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.279210    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:41:03.279531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:03.279355    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2p7c5\""
Jul 10 18:41:05.860978 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:05.860935    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-x8cj5"]
Jul 10 18:41:05.881825 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:05.881785    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:05.909676 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:05.909646    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-x8cj5"]
Jul 10 18:41:05.909827 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:05.909759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-x8cj5"
Jul 10 18:41:05.914495 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:05.914472    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-khxz6\""
Jul 10 18:41:05.915054 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:05.915034    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 10 18:41:05.915242 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:05.915226    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 10 18:41:05.982511 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:05.982466    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.062087 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.062045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsvm\" (UniqueName: \"kubernetes.io/projected/a51a24dc-e29e-4dd7-a621-4c169d6d282d-kube-api-access-bhsvm\") pod \"downloads-68c9cbb497-x8cj5\" (UID: \"a51a24dc-e29e-4dd7-a621-4c169d6d282d\") " pod="openshift-console/downloads-68c9cbb497-x8cj5"
Jul 10 18:41:06.135864 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.135780    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.162802 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.162767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsvm\" (UniqueName: \"kubernetes.io/projected/a51a24dc-e29e-4dd7-a621-4c169d6d282d-kube-api-access-bhsvm\") pod \"downloads-68c9cbb497-x8cj5\" (UID: \"a51a24dc-e29e-4dd7-a621-4c169d6d282d\") " pod="openshift-console/downloads-68c9cbb497-x8cj5"
Jul 10 18:41:06.167469 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.167439    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8jxkm"]
Jul 10 18:41:06.187187 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.187163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsvm\" (UniqueName: \"kubernetes.io/projected/a51a24dc-e29e-4dd7-a621-4c169d6d282d-kube-api-access-bhsvm\") pod \"downloads-68c9cbb497-x8cj5\" (UID: \"a51a24dc-e29e-4dd7-a621-4c169d6d282d\") " pod="openshift-console/downloads-68c9cbb497-x8cj5"
Jul 10 18:41:06.220526 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.219982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-x8cj5"
Jul 10 18:41:06.221014 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.220990    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8jxkm"]
Jul 10 18:41:06.221124 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.221080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.224325 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.224296    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 18:41:06.224447 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.224375    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 10 18:41:06.224518 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.224483    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 18:41:06.225201 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.225186    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 10 18:41:06.225380 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.225359    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 18:41:06.225588 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.225573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7svzh\""
Jul 10 18:41:06.233657 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.233632    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.346836 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.346809    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.364527 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.364490    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.364681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.364598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b132a3f7-4f59-4159-9411-8e3562002c89-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.364681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.364623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.364681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.364655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xh62\" (UniqueName: \"kubernetes.io/projected/b132a3f7-4f59-4159-9411-8e3562002c89-kube-api-access-7xh62\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.466019 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.465987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b132a3f7-4f59-4159-9411-8e3562002c89-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.466219 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.466031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.466219 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.466066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xh62\" (UniqueName: \"kubernetes.io/projected/b132a3f7-4f59-4159-9411-8e3562002c89-kube-api-access-7xh62\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.466219 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.466092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.466449 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:06.466227    2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 10 18:41:06.466449 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:06.466356    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-tls podName:b132a3f7-4f59-4159-9411-8e3562002c89 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:06.966330835 +0000 UTC m=+48.264672883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-tls") pod "prometheus-operator-64785b6675-8jxkm" (UID: "b132a3f7-4f59-4159-9411-8e3562002c89") : secret "prometheus-operator-tls" not found
Jul 10 18:41:06.466749 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.466727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b132a3f7-4f59-4159-9411-8e3562002c89-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.469008 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.468979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.483521 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.483499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xh62\" (UniqueName: \"kubernetes.io/projected/b132a3f7-4f59-4159-9411-8e3562002c89-kube-api-access-7xh62\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.503535 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.503501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.604162 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.604131    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.675768 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.675734    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:06.971066 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.971027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:06.973464 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:06.973441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b132a3f7-4f59-4159-9411-8e3562002c89-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8jxkm\" (UID: \"b132a3f7-4f59-4159-9411-8e3562002c89\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:07.131457 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:07.131418    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm"
Jul 10 18:41:07.560815 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:07.560786    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:07.656371 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:07.656335    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:07.757431 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:07.757157    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:07.945346 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:07.945280    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:08.178647 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:08.178616    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:08.324736 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:08.324518    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8jxkm"]
Jul 10 18:41:08.328675 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:08.328386    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-x8cj5"]
Jul 10 18:41:08.401835 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:08.401799    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb132a3f7_4f59_4159_9411_8e3562002c89.slice/crio-830b395a1e21a2e830074efb45af1c2f552bc4824be1401b15dc4a51c698dd27 WatchSource:0}: Error finding container 830b395a1e21a2e830074efb45af1c2f552bc4824be1401b15dc4a51c698dd27: Status 404 returned error can't find the container with id 830b395a1e21a2e830074efb45af1c2f552bc4824be1401b15dc4a51c698dd27
Jul 10 18:41:08.402107 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:08.402075    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51a24dc_e29e_4dd7_a621_4c169d6d282d.slice/crio-eea27d8c5a85661bcb52b99ecc264f76679466911eb30a7e31655c30b5913c09 WatchSource:0}: Error finding container eea27d8c5a85661bcb52b99ecc264f76679466911eb30a7e31655c30b5913c09: Status 404 returned error can't find the container with id eea27d8c5a85661bcb52b99ecc264f76679466911eb30a7e31655c30b5913c09
Jul 10 18:41:08.492894 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:08.492868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-x8cj5" event={"ID":"a51a24dc-e29e-4dd7-a621-4c169d6d282d","Type":"ContainerStarted","Data":"eea27d8c5a85661bcb52b99ecc264f76679466911eb30a7e31655c30b5913c09"}
Jul 10 18:41:08.494198 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:08.494174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2pk2" event={"ID":"c4f27378-5bf0-4d9a-9508-06c76b584cdc","Type":"ContainerStarted","Data":"d40cb23914662c2ff5da149565b0732cefc8a8b2b2a4113ac462ea90ba3ee4c7"}
Jul 10 18:41:08.495549 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:08.495530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nxzsc" event={"ID":"fd2847f5-9eae-4319-a9e4-061d9d3f2da1","Type":"ContainerStarted","Data":"79f10ac91bdea2a182435b2b43c2d4397495853a57f48d09e2a2654bcd38e5d7"}
Jul 10 18:41:08.496567 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:08.496550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm" event={"ID":"b132a3f7-4f59-4159-9411-8e3562002c89","Type":"ContainerStarted","Data":"830b395a1e21a2e830074efb45af1c2f552bc4824be1401b15dc4a51c698dd27"}
Jul 10 18:41:08.518275 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:08.517998    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nxzsc" podStartSLOduration=1.57603694 podStartE2EDuration="7.517945062s" totalImagesPullingTime="5.941908122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:08.516888929 +0000 UTC m=+49.815231001" watchObservedRunningTime="2026-07-10 18:41:08.517945062 +0000 UTC m=+49.816287133"
Jul 10 18:41:09.060405 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:09.060365    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:09.149650 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:09.149615    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:09.501333 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:09.501120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2pk2" event={"ID":"c4f27378-5bf0-4d9a-9508-06c76b584cdc","Type":"ContainerStarted","Data":"2ef8bf0e9d9948d7b89fd2f2380c8ef80a9b28a03586a0c6ee5e29af73568d3a"}
Jul 10 18:41:09.502146 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:09.501363    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:09.502557 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:09.502538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5f954" event={"ID":"c02d7e07-698b-4351-9db2-0b570b3b8570","Type":"ContainerStarted","Data":"be761bec200b2b765b79552f36640c24cb83c83961bfb793ff12455d79dab700"}
Jul 10 18:41:09.524572 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:09.524530    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z2pk2" podStartSLOduration=2.588971673 podStartE2EDuration="8.524514457s" totalImagesPullingTime="5.935542784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:09.523389438 +0000 UTC m=+50.821731508" watchObservedRunningTime="2026-07-10 18:41:09.524514457 +0000 UTC m=+50.822856525"
Jul 10 18:41:10.507329 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:10.507290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5f954" event={"ID":"c02d7e07-698b-4351-9db2-0b570b3b8570","Type":"ContainerStarted","Data":"abaeeca09409a5e0c791fbf9f9dbc918256e757568db26571a26e69426d3f551"}
Jul 10 18:41:11.247560 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:11.247359    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:11.514655 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:11.514567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm" event={"ID":"b132a3f7-4f59-4159-9411-8e3562002c89","Type":"ContainerStarted","Data":"edd94bbf6e6a3aec0581f49e190cc428ce675a8abce0a1b89309d92ae3a7f005"}
Jul 10 18:41:11.514655 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:11.514637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm" event={"ID":"b132a3f7-4f59-4159-9411-8e3562002c89","Type":"ContainerStarted","Data":"f73e0516e8579be1faeeb36b683eb0d8ddbedab28371835e2a76c11a1f64f632"}
Jul 10 18:41:11.537247 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:11.537175    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-8jxkm" podStartSLOduration=3.09099879 podStartE2EDuration="5.537153699s" totalImagesPullingTime="2.446154909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:11.536007219 +0000 UTC m=+52.834349315" watchObservedRunningTime="2026-07-10 18:41:11.537153699 +0000 UTC m=+52.835495770"
Jul 10 18:41:12.476067 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:12.476031    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:12.575896 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:12.575855    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.259756 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.259721    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.388564 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.388532    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.523981 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.523831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5f954" event={"ID":"c02d7e07-698b-4351-9db2-0b570b3b8570","Type":"ContainerStarted","Data":"aa0566d28f920b3cd60f20a57b2b95afc00891659c52b313b051ccebcc9a4529"}
Jul 10 18:41:13.555337 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.555290    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5f954" podStartSLOduration=1.81755134 podStartE2EDuration="12.555272185s" totalImagesPullingTime="10.737720845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:41:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:13.554399459 +0000 UTC m=+54.852741540" watchObservedRunningTime="2026-07-10 18:41:13.555272185 +0000 UTC m=+54.853614247"
Jul 10 18:41:13.570341 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.570312    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.606988 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.606934    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"]
Jul 10 18:41:13.609186 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.609163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.611609 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.611586    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lsl7j\""
Jul 10 18:41:13.612088 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.612069    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 10 18:41:13.612190 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.612069    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 10 18:41:13.614243 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.614215    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"]
Jul 10 18:41:13.616451 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.616389    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.621014 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.620996    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 10 18:41:13.621111 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.621049    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 10 18:41:13.621237 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.621224    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 10 18:41:13.621374 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.621350    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5rfsw\""
Jul 10 18:41:13.622669 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.622647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.622779 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.622748    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d68bbe0-dc50-45bc-9176-2603887ce446-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.622837 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.622775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.622837 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.622801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.622837 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.622827    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m2kn\" (UniqueName: \"kubernetes.io/projected/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-kube-api-access-5m2kn\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.622939 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.622870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.622939 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.622899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.623063 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.622987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wnn\" (UniqueName: \"kubernetes.io/projected/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-api-access-82wnn\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.623063 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.623010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.623063 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.623038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d68bbe0-dc50-45bc-9176-2603887ce446-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.631997 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.631974    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"]
Jul 10 18:41:13.633791 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.633772    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lvdjd"]
Jul 10 18:41:13.636015 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.635996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.637746 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.637688    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"]
Jul 10 18:41:13.638715 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.638472    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2kssn\""
Jul 10 18:41:13.638715 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.638495    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 18:41:13.638854 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.638745    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 18:41:13.639185 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.639165    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 18:41:13.679712 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.679684    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.723684 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-sys\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.723684 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-textfile\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.723917 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.723917 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-root\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.723917 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-tls\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.723917 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wnn\" (UniqueName: \"kubernetes.io/projected/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-api-access-82wnn\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.723917 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7j29\" (UniqueName: \"kubernetes.io/projected/eacdb005-b9a5-41af-a3a5-25df81aa41a3-kube-api-access-m7j29\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.723917 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-wtmp\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.723917 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m2kn\" (UniqueName: \"kubernetes.io/projected/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-kube-api-access-5m2kn\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.723979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eacdb005-b9a5-41af-a3a5-25df81aa41a3-metrics-client-ca\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d68bbe0-dc50-45bc-9176-2603887ce446-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.724280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d68bbe0-dc50-45bc-9176-2603887ce446-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.724591 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.724537    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d68bbe0-dc50-45bc-9176-2603887ce446-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.725923 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:13.724847    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 10 18:41:13.725923 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:13.724925    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-tls podName:7d68bbe0-dc50-45bc-9176-2603887ce446 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:14.224903621 +0000 UTC m=+55.523245682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-bhp9q" (UID: "7d68bbe0-dc50-45bc-9176-2603887ce446") : secret "kube-state-metrics-tls" not found
Jul 10 18:41:13.725923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.725507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.725923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.725520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d68bbe0-dc50-45bc-9176-2603887ce446-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.725923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.725860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.728542 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.728519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.728542 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.728530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.728542 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.728537    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.739502 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.739480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m2kn\" (UniqueName: \"kubernetes.io/projected/7dccc0b5-9d9e-46c4-8255-5c874c289bfc-kube-api-access-5m2kn\") pod \"openshift-state-metrics-84bd68b896-n8xrf\" (UID: \"7dccc0b5-9d9e-46c4-8255-5c874c289bfc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.739684 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.739659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wnn\" (UniqueName: \"kubernetes.io/projected/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-api-access-82wnn\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:13.825060 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.824940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825060 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-sys\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825060 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-textfile\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825060 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-root\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-tls\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7j29\" (UniqueName: \"kubernetes.io/projected/eacdb005-b9a5-41af-a3a5-25df81aa41a3-kube-api-access-m7j29\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-wtmp\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-root\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-sys\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825277    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-wtmp\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eacdb005-b9a5-41af-a3a5-25df81aa41a3-metrics-client-ca\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825647 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-textfile\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825752 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.825791 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.825735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eacdb005-b9a5-41af-a3a5-25df81aa41a3-metrics-client-ca\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.827624 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.827601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.827718 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.827697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eacdb005-b9a5-41af-a3a5-25df81aa41a3-node-exporter-tls\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.833384 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.833364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7j29\" (UniqueName: \"kubernetes.io/projected/eacdb005-b9a5-41af-a3a5-25df81aa41a3-kube-api-access-m7j29\") pod \"node-exporter-lvdjd\" (UID: \"eacdb005-b9a5-41af-a3a5-25df81aa41a3\") " pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.848681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.848657    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.920173 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.920138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"
Jul 10 18:41:13.945536 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.945477    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:13.946915 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:13.946890    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvdjd"
Jul 10 18:41:13.958646 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:13.958620    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacdb005_b9a5_41af_a3a5_25df81aa41a3.slice/crio-c50a8d9c30b1bbd5f850dd15350b6612c225c1df4ee5785e7fd6896479c034e4 WatchSource:0}: Error finding container c50a8d9c30b1bbd5f850dd15350b6612c225c1df4ee5785e7fd6896479c034e4: Status 404 returned error can't find the container with id c50a8d9c30b1bbd5f850dd15350b6612c225c1df4ee5785e7fd6896479c034e4
Jul 10 18:41:14.072787 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.072606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf"]
Jul 10 18:41:14.076464 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.076404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:14.076779 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:14.076755    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dccc0b5_9d9e_46c4_8255_5c874c289bfc.slice/crio-5cd25dc62c8ca166444abbda736aea3d5c0324e54c2d41e42b93126800029f51 WatchSource:0}: Error finding container 5cd25dc62c8ca166444abbda736aea3d5c0324e54c2d41e42b93126800029f51: Status 404 returned error can't find the container with id 5cd25dc62c8ca166444abbda736aea3d5c0324e54c2d41e42b93126800029f51
Jul 10 18:41:14.185999 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.185890    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:14.228362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.228333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:14.234273 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.234236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d68bbe0-dc50-45bc-9176-2603887ce446-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-bhp9q\" (UID: \"7d68bbe0-dc50-45bc-9176-2603887ce446\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:14.528628 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.528596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"
Jul 10 18:41:14.529975 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.529922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvdjd" event={"ID":"eacdb005-b9a5-41af-a3a5-25df81aa41a3","Type":"ContainerStarted","Data":"c50a8d9c30b1bbd5f850dd15350b6612c225c1df4ee5785e7fd6896479c034e4"}
Jul 10 18:41:14.533039 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.533001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf" event={"ID":"7dccc0b5-9d9e-46c4-8255-5c874c289bfc","Type":"ContainerStarted","Data":"4565140d6be53d04214878a0d94c53a2109cf25f10df71531775112a1f0acbbb"}
Jul 10 18:41:14.533130 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.533076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf" event={"ID":"7dccc0b5-9d9e-46c4-8255-5c874c289bfc","Type":"ContainerStarted","Data":"adbfeb176d231a7486ed528ae6f3cac3c68fb3d3590d48c3eb201a849f7a100b"}
Jul 10 18:41:14.533130 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.533108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf" event={"ID":"7dccc0b5-9d9e-46c4-8255-5c874c289bfc","Type":"ContainerStarted","Data":"5cd25dc62c8ca166444abbda736aea3d5c0324e54c2d41e42b93126800029f51"}
Jul 10 18:41:14.553773 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.553697    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:14.668367 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.668322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:14.707970 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:14.707916    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-bhp9q"]
Jul 10 18:41:14.711549 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:14.711524    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d68bbe0_dc50_45bc_9176_2603887ce446.slice/crio-40134d69ee7381954fba3927e9d5524ec644122b800b0016b1edf8267c0555a8 WatchSource:0}: Error finding container 40134d69ee7381954fba3927e9d5524ec644122b800b0016b1edf8267c0555a8: Status 404 returned error can't find the container with id 40134d69ee7381954fba3927e9d5524ec644122b800b0016b1edf8267c0555a8
Jul 10 18:41:15.003924 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.003651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:15.079436 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.079409    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:15.180256 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.180220    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:15.538027 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.537988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q" event={"ID":"7d68bbe0-dc50-45bc-9176-2603887ce446","Type":"ContainerStarted","Data":"40134d69ee7381954fba3927e9d5524ec644122b800b0016b1edf8267c0555a8"}
Jul 10 18:41:15.539727 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.539699    2565 generic.go:356] "Generic (PLEG): container finished" podID="eacdb005-b9a5-41af-a3a5-25df81aa41a3" containerID="a22627d5c5157f4c1412e8994698a73c28dd3d0cde9c15841e35c60de447436d" exitCode=0
Jul 10 18:41:15.539879 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.539801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvdjd" event={"ID":"eacdb005-b9a5-41af-a3a5-25df81aa41a3","Type":"ContainerDied","Data":"a22627d5c5157f4c1412e8994698a73c28dd3d0cde9c15841e35c60de447436d"}
Jul 10 18:41:15.542610 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.542584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf" event={"ID":"7dccc0b5-9d9e-46c4-8255-5c874c289bfc","Type":"ContainerStarted","Data":"74f378665e3ac464950ea46e06bafc556449caa6fbb80812dad223dfffa18d44"}
Jul 10 18:41:15.581359 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.581255    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n8xrf" podStartSLOduration=1.429207103 podStartE2EDuration="2.581232856s" totalImagesPullingTime="1.152025753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:15.580489861 +0000 UTC m=+56.878831931" watchObservedRunningTime="2026-07-10 18:41:15.581232856 +0000 UTC m=+56.879574926"
Jul 10 18:41:15.817904 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.817835    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:15.881265 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:15.881182    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:16.516192 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.516161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:16.548460 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.548426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvdjd" event={"ID":"eacdb005-b9a5-41af-a3a5-25df81aa41a3","Type":"ContainerStarted","Data":"5a25531dbb4a4a35c8a1628b5565fe5c355452f332bc61fd5d411c97eda28721"}
Jul 10 18:41:16.548628 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.548491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvdjd" event={"ID":"eacdb005-b9a5-41af-a3a5-25df81aa41a3","Type":"ContainerStarted","Data":"de10f547961fff422ffc1b7171d77efe128cda9b9649585b60cb1bceecf1ac34"}
Jul 10 18:41:16.551470 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.551441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q" event={"ID":"7d68bbe0-dc50-45bc-9176-2603887ce446","Type":"ContainerStarted","Data":"fa1b3ddea8b37197be17a0c6eed4cb4757cc8c153c5d18da2d90f903330f7d09"}
Jul 10 18:41:16.574107 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.574006    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lvdjd" podStartSLOduration=2.403576513 podStartE2EDuration="3.573988636s" totalImagesPullingTime="1.090047029s" totalInitContainerRuntime="80.365094ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:16.572778732 +0000 UTC m=+57.871120822" watchObservedRunningTime="2026-07-10 18:41:16.573988636 +0000 UTC m=+57.872330704"
Jul 10 18:41:16.620508 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.618720    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:16.625990 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.624758    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"]
Jul 10 18:41:16.629411 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.628603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.633551 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.633503    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vxb8b\""
Jul 10 18:41:16.636709 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.633761    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 10 18:41:16.636709 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.634542    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 10 18:41:16.636709 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.635775    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 10 18:41:16.636709 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.636021    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b24lj3dl5p4ra\""
Jul 10 18:41:16.637668 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.637647    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 10 18:41:16.637875 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.637858    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 10 18:41:16.647986 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.647593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"]
Jul 10 18:41:16.650825 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.650709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.650825 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.650752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.650825 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.650791    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.651118 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.650874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b9e1de-ad0c-40e7-98af-d4664074ca21-metrics-client-ca\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.651118 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.651034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-tls\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.651220 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.651158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-grpc-tls\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.653679 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.653654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpg78\" (UniqueName: \"kubernetes.io/projected/70b9e1de-ad0c-40e7-98af-d4664074ca21-kube-api-access-kpg78\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.653908 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.653890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.732790 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.732721    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:16.754666 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.754635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-tls\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.754858 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.754696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-grpc-tls\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.754858 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.754724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpg78\" (UniqueName: \"kubernetes.io/projected/70b9e1de-ad0c-40e7-98af-d4664074ca21-kube-api-access-kpg78\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.755113 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.755092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.755171 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.755147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.755225 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.755176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.755275 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.755221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.755275 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.755254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b9e1de-ad0c-40e7-98af-d4664074ca21-metrics-client-ca\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.756067 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.756027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b9e1de-ad0c-40e7-98af-d4664074ca21-metrics-client-ca\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.760358 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.760329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.760517 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.760478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.761291 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.761259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.761822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.761796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-tls\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.761899 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.761820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-grpc-tls\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.762746 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.762724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/70b9e1de-ad0c-40e7-98af-d4664074ca21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.764927 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.764837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpg78\" (UniqueName: \"kubernetes.io/projected/70b9e1de-ad0c-40e7-98af-d4664074ca21-kube-api-access-kpg78\") pod \"thanos-querier-bdbbc5c54-7hgkl\" (UID: \"70b9e1de-ad0c-40e7-98af-d4664074ca21\") " pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:16.969173 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:16.968802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:17.020606 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:17.020479    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:17.131109 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:17.131080    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"]
Jul 10 18:41:17.133649 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:17.133618    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b9e1de_ad0c_40e7_98af_d4664074ca21.slice/crio-44fc2e228ee918599c622d21ce206a75fa349637bbb5202c7e789683628456e9 WatchSource:0}: Error finding container 44fc2e228ee918599c622d21ce206a75fa349637bbb5202c7e789683628456e9: Status 404 returned error can't find the container with id 44fc2e228ee918599c622d21ce206a75fa349637bbb5202c7e789683628456e9
Jul 10 18:41:17.556681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:17.556639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl" event={"ID":"70b9e1de-ad0c-40e7-98af-d4664074ca21","Type":"ContainerStarted","Data":"44fc2e228ee918599c622d21ce206a75fa349637bbb5202c7e789683628456e9"}
Jul 10 18:41:17.559030 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:17.558993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q" event={"ID":"7d68bbe0-dc50-45bc-9176-2603887ce446","Type":"ContainerStarted","Data":"4e720464b3ecfa74230ec3781b8f45bbbb156998f1e5ac75b0ad4302eb64121e"}
Jul 10 18:41:17.559175 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:17.559060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q" event={"ID":"7d68bbe0-dc50-45bc-9176-2603887ce446","Type":"ContainerStarted","Data":"7f33c062ca90bf2dea71c149b7e229cbb244d9597d5ad03ac5901d8bc788de43"}
Jul 10 18:41:17.581722 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:17.581659    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-bhp9q" podStartSLOduration=2.867861145 podStartE2EDuration="4.581636245s" totalImagesPullingTime="1.7137751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:17.580297282 +0000 UTC m=+58.878639374" watchObservedRunningTime="2026-07-10 18:41:17.581636245 +0000 UTC m=+58.879978316"
Jul 10 18:41:18.304792 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.304764    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:18.315972 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.315918    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"]
Jul 10 18:41:18.331175 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.331135    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:18.337647 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.337620    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dcjqc\""
Jul 10 18:41:18.337836 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.337808    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 10 18:41:18.338314 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.338293    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"]
Jul 10 18:41:18.372394 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.372350    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/223dc601-b63a-4b63-ae11-091f6ae6f6a7-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-x96hd\" (UID: \"223dc601-b63a-4b63-ae11-091f6ae6f6a7\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:18.372609 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.372426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdjlh\" (UniqueName: \"kubernetes.io/projected/223dc601-b63a-4b63-ae11-091f6ae6f6a7-kube-api-access-tdjlh\") pod \"monitoring-plugin-5f68df7f58-x96hd\" (UID: \"223dc601-b63a-4b63-ae11-091f6ae6f6a7\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:18.395274 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.395243    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:18.444485 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.444435    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l79f7"
Jul 10 18:41:18.473399 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.473360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/223dc601-b63a-4b63-ae11-091f6ae6f6a7-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-x96hd\" (UID: \"223dc601-b63a-4b63-ae11-091f6ae6f6a7\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:18.473580 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.473409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdjlh\" (UniqueName: \"kubernetes.io/projected/223dc601-b63a-4b63-ae11-091f6ae6f6a7-kube-api-access-tdjlh\") pod \"monitoring-plugin-5f68df7f58-x96hd\" (UID: \"223dc601-b63a-4b63-ae11-091f6ae6f6a7\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:18.473580 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:18.473514    2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 10 18:41:18.473689 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:41:18.473593    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223dc601-b63a-4b63-ae11-091f6ae6f6a7-monitoring-plugin-cert podName:223dc601-b63a-4b63-ae11-091f6ae6f6a7 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:18.973572 +0000 UTC m=+60.271914072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/223dc601-b63a-4b63-ae11-091f6ae6f6a7-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-x96hd" (UID: "223dc601-b63a-4b63-ae11-091f6ae6f6a7") : secret "monitoring-plugin-cert" not found
Jul 10 18:41:18.486896 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.486867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdjlh\" (UniqueName: \"kubernetes.io/projected/223dc601-b63a-4b63-ae11-091f6ae6f6a7-kube-api-access-tdjlh\") pod \"monitoring-plugin-5f68df7f58-x96hd\" (UID: \"223dc601-b63a-4b63-ae11-091f6ae6f6a7\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:18.580737 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.580656    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:18.682387 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.682351    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:18.979113 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.979075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/223dc601-b63a-4b63-ae11-091f6ae6f6a7-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-x96hd\" (UID: \"223dc601-b63a-4b63-ae11-091f6ae6f6a7\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:18.981623 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:18.981599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/223dc601-b63a-4b63-ae11-091f6ae6f6a7-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-x96hd\" (UID: \"223dc601-b63a-4b63-ae11-091f6ae6f6a7\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:19.001123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.001085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:19.247114 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.247026    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dcjqc\""
Jul 10 18:41:19.254786 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.254752    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:19.510539 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.510456    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z2pk2"
Jul 10 18:41:19.939757 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.939717    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:41:19.945811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.945771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.951222 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.950887    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wdln6\""
Jul 10 18:41:19.952233 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.952155    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 10 18:41:19.952351 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.952311    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 10 18:41:19.953011 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.952827    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 10 18:41:19.953682 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.953286    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 10 18:41:19.953682 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.953494    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 10 18:41:19.956871 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.956814    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.957395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.957447    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.957485    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.957706    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.958070    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ci3nnqgglsdsm\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.958110    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.958338    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.958339    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.958739    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 10 18:41:19.959204 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.958912    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 10 18:41:19.977120 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.977036    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:19.988044 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.987300    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:41:19.998778 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.998778 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.998778 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.998778 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.998778 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-config\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.998778 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.998991    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999133 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999551 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999551 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999551 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999551 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999241    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:19.999551 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:19.999270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shxs\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-kube-api-access-4shxs\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.007776 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.007736    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"]
Jul 10 18:41:20.100435 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.100405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxs\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-kube-api-access-4shxs\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.100574 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.100469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.100574 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.100509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.100574 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.100537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.100744 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.100581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.100744 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.100604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-config\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.100744 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.100632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101183 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101183 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101286 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101286 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101286 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101286 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101474 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101474 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101474 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101474 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101474 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.101474 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.101470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.102237 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.102213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.102592 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.102568    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.103084 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.103057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.103586 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.103561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.104262 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.104236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.104537 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.104505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.105446 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.104856    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.105446 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.104885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.105446 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.105085    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.105446 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.105419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-config\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.105705 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.105677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.106641 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.106586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.106923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.106897    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.107027 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.106931    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.107089 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.106941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.108138 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.108107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.108253 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.108236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.109124 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.109071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.110527 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.110509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.113707 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.113684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shxs\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-kube-api-access-4shxs\") pod \"prometheus-k8s-0\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.262390 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.262352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:20.453048 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.452933    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:41:20.456923 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:20.456894    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e914c9b_6f51_4175_882e_3b184640fc45.slice/crio-c2a091bb6c37d3293e15cabdfbb688bcb23a56f2de9450152b758e8b41c99376 WatchSource:0}: Error finding container c2a091bb6c37d3293e15cabdfbb688bcb23a56f2de9450152b758e8b41c99376: Status 404 returned error can't find the container with id c2a091bb6c37d3293e15cabdfbb688bcb23a56f2de9450152b758e8b41c99376
Jul 10 18:41:20.579294 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.579097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl" event={"ID":"70b9e1de-ad0c-40e7-98af-d4664074ca21","Type":"ContainerStarted","Data":"958c3d7b71330ddba0d90d3ac2a525aa7ffdc7acc2280828f539b9ac22e3cb40"}
Jul 10 18:41:20.579294 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.579186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl" event={"ID":"70b9e1de-ad0c-40e7-98af-d4664074ca21","Type":"ContainerStarted","Data":"fc18b2f2d64f4f6a108bf01320a0823157e23da0aacd142fdaa014585baef685"}
Jul 10 18:41:20.579294 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.579238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl" event={"ID":"70b9e1de-ad0c-40e7-98af-d4664074ca21","Type":"ContainerStarted","Data":"bc753b003a68867d7110c5049cbddc4947cdd00711c9b1e4e3fd847abd1bd5f2"}
Jul 10 18:41:20.581308 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.581270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd" event={"ID":"223dc601-b63a-4b63-ae11-091f6ae6f6a7","Type":"ContainerStarted","Data":"7fdd1e68761c502b0bd45983345ddc842fbf11869af2a89756f9701cfa6bc4ae"}
Jul 10 18:41:20.583828 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:20.583801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerStarted","Data":"c2a091bb6c37d3293e15cabdfbb688bcb23a56f2de9450152b758e8b41c99376"}
Jul 10 18:41:21.158212 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.158148    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:21.259288 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.259200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:21.591712 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.591627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl" event={"ID":"70b9e1de-ad0c-40e7-98af-d4664074ca21","Type":"ContainerStarted","Data":"75db7ada0dd978754d615976b164fbdeea2bdb02649956c37b817b382452aa2c"}
Jul 10 18:41:21.592163 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.591712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl" event={"ID":"70b9e1de-ad0c-40e7-98af-d4664074ca21","Type":"ContainerStarted","Data":"e313418229c687dcde7438ec34e4ca52f6575527b85fa4d5ed7aa29d297c3b35"}
Jul 10 18:41:21.592163 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.591765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl" event={"ID":"70b9e1de-ad0c-40e7-98af-d4664074ca21","Type":"ContainerStarted","Data":"0bad7b987b9d71df0c6ddfa8c0d62b168e973da8920ecff50c341616da7ae203"}
Jul 10 18:41:21.592163 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.591997    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:21.624386 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.624317    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl" podStartSLOduration=2.095975092 podStartE2EDuration="5.624293132s" totalImagesPullingTime="3.52831804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:21.622915427 +0000 UTC m=+62.921257538" watchObservedRunningTime="2026-07-10 18:41:21.624293132 +0000 UTC m=+62.922635204"
Jul 10 18:41:21.736702 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.736666    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:21.837253 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:21.837218    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:22.599063 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:22.599020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd" event={"ID":"223dc601-b63a-4b63-ae11-091f6ae6f6a7","Type":"ContainerStarted","Data":"af60cd7b5ff3050e476834e19df8d8c9dd4164283e2000b3ed5dea7a5dcb86a9"}
Jul 10 18:41:22.599607 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:22.599462    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:22.600910 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:22.600882    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e914c9b-6f51-4175-882e-3b184640fc45" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7" exitCode=0
Jul 10 18:41:22.601071 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:22.600988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerDied","Data":"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"}
Jul 10 18:41:22.605638 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:22.605613    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd"
Jul 10 18:41:22.622283 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:22.622221    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x96hd" podStartSLOduration=2.776255334 podStartE2EDuration="4.622199025s" totalImagesPullingTime="1.845943691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:22.619982924 +0000 UTC m=+63.918325178" watchObservedRunningTime="2026-07-10 18:41:22.622199025 +0000 UTC m=+63.920541098"
Jul 10 18:41:22.834482 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:22.834441    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:22.913855 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:22.913821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:23.202577 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:23.202433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:23.301635 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:23.301608    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:24.049299 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.049246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:41:24.049299 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.049316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:41:24.051997 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.051975    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:41:24.052828 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.052812    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:41:24.062466 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.062434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0e8e075-a99c-4edf-b1ba-3a5504c6734d-original-pull-secret\") pod \"global-pull-secret-syncer-2m65d\" (UID: \"d0e8e075-a99c-4edf-b1ba-3a5504c6734d\") " pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:41:24.062605 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.062473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1100f38-e07c-440d-b610-7f2dc8035c9c-metrics-certs\") pod \"network-metrics-daemon-vhp8k\" (UID: \"f1100f38-e07c-440d-b610-7f2dc8035c9c\") " pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:41:24.150321 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.150277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:41:24.153057 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.153036    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:41:24.163174 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.163151    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:41:24.173900 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.173873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzsk\" (UniqueName: \"kubernetes.io/projected/3be1ba28-5b3d-4f64-9575-5e98a563732d-kube-api-access-ctzsk\") pod \"network-check-target-vfbvz\" (UID: \"3be1ba28-5b3d-4f64-9575-5e98a563732d\") " pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:41:24.292547 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.292504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m65d"
Jul 10 18:41:24.301132 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.301056    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2p7c5\""
Jul 10 18:41:24.308410 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.308389    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6dtsl\""
Jul 10 18:41:24.309406 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.309384    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhp8k"
Jul 10 18:41:24.316452 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:24.316428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:41:25.185595 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:25.185566    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:25.285142 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:25.285109    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:25.384601 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:25.384566    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:25.456060 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:25.455974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:26.756479 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:26.756347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:26.863784 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:26.863740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:27.608617 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:27.608585    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bdbbc5c54-7hgkl"
Jul 10 18:41:28.266521 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:28.266488    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:28.390342 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:28.390305    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:29.747695 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:29.747667    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vfbvz"]
Jul 10 18:41:29.750934 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:29.750103    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be1ba28_5b3d_4f64_9575_5e98a563732d.slice/crio-78a575f9d5c2af29dfbe9b228980c833499e79d6d9b0cb10d88663de558bcf38 WatchSource:0}: Error finding container 78a575f9d5c2af29dfbe9b228980c833499e79d6d9b0cb10d88663de558bcf38: Status 404 returned error can't find the container with id 78a575f9d5c2af29dfbe9b228980c833499e79d6d9b0cb10d88663de558bcf38
Jul 10 18:41:29.973854 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:29.973714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vhp8k"]
Jul 10 18:41:29.976638 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:29.976593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2m65d"]
Jul 10 18:41:29.976948 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:29.976921    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1100f38_e07c_440d_b610_7f2dc8035c9c.slice/crio-cfb03f6b0090574da431a7a5ed5a40e949218b34517082e6cb465e7db4a7b0a7 WatchSource:0}: Error finding container cfb03f6b0090574da431a7a5ed5a40e949218b34517082e6cb465e7db4a7b0a7: Status 404 returned error can't find the container with id cfb03f6b0090574da431a7a5ed5a40e949218b34517082e6cb465e7db4a7b0a7
Jul 10 18:41:29.980918 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:41:29.980889    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e8e075_a99c_4edf_b1ba_3a5504c6734d.slice/crio-471ab0f1457eae578839d63ad9f7f984fc7620b8575e6efb28f3bc56ae5544a3 WatchSource:0}: Error finding container 471ab0f1457eae578839d63ad9f7f984fc7620b8575e6efb28f3bc56ae5544a3: Status 404 returned error can't find the container with id 471ab0f1457eae578839d63ad9f7f984fc7620b8575e6efb28f3bc56ae5544a3
Jul 10 18:41:30.637086 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:30.637046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vfbvz" event={"ID":"3be1ba28-5b3d-4f64-9575-5e98a563732d","Type":"ContainerStarted","Data":"78a575f9d5c2af29dfbe9b228980c833499e79d6d9b0cb10d88663de558bcf38"}
Jul 10 18:41:30.639206 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:30.639115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vhp8k" event={"ID":"f1100f38-e07c-440d-b610-7f2dc8035c9c","Type":"ContainerStarted","Data":"cfb03f6b0090574da431a7a5ed5a40e949218b34517082e6cb465e7db4a7b0a7"}
Jul 10 18:41:30.642065 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:30.642034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-x8cj5" event={"ID":"a51a24dc-e29e-4dd7-a621-4c169d6d282d","Type":"ContainerStarted","Data":"d906ef908e1d43e32f8cdb7d9999a6dbe27a1ae036c4aed289782b6d39176e26"}
Jul 10 18:41:30.643388 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:30.643363    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-x8cj5"
Jul 10 18:41:30.646885 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:30.646822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2m65d" event={"ID":"d0e8e075-a99c-4edf-b1ba-3a5504c6734d","Type":"ContainerStarted","Data":"471ab0f1457eae578839d63ad9f7f984fc7620b8575e6efb28f3bc56ae5544a3"}
Jul 10 18:41:30.655429 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:30.655383    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-x8cj5"
Jul 10 18:41:30.665802 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:30.662932    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-x8cj5" podStartSLOduration=3.945672227 podStartE2EDuration="25.662912662s" totalImagesPullingTime="21.717240435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:30.661996473 +0000 UTC m=+71.960338565" watchObservedRunningTime="2026-07-10 18:41:30.662912662 +0000 UTC m=+71.961254729"
Jul 10 18:41:30.900339 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:30.900215    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:31.014822 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:31.014754    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:31.937148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:31.937055    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:32.024048 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:32.023925    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:32.675963 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:32.675913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vhp8k" event={"ID":"f1100f38-e07c-440d-b610-7f2dc8035c9c","Type":"ContainerStarted","Data":"e8bbcc03c6cffe5e46c9eacd34aad488b394000ef0982b87ecfc8e0596347110"}
Jul 10 18:41:32.676107 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:32.676031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vhp8k" event={"ID":"f1100f38-e07c-440d-b610-7f2dc8035c9c","Type":"ContainerStarted","Data":"ecda000e55fd60cfbc48bec188cc0cc01af67af90b85db8dbd53f9434117dca8"}
Jul 10 18:41:32.697635 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:32.697572    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vhp8k" podStartSLOduration=71.427819542 podStartE2EDuration="1m13.697548434s" totalImagesPullingTime="2.269728892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:32.695498431 +0000 UTC m=+73.993840500" watchObservedRunningTime="2026-07-10 18:41:32.697548434 +0000 UTC m=+73.995890559"
Jul 10 18:41:32.819115 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:32.818982    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:32.905934 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:32.905809    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:38.709301 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:38.708807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vfbvz" event={"ID":"3be1ba28-5b3d-4f64-9575-5e98a563732d","Type":"ContainerStarted","Data":"822c7ef5d9c8647dfdc28accdbd770bc2acd0a57af33f1f8120421beb9df9831"}
Jul 10 18:41:38.709301 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:38.709108    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:41:38.713237 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:38.713212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerStarted","Data":"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"}
Jul 10 18:41:38.724788 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:38.724739    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vfbvz" podStartSLOduration=70.913037332 podStartE2EDuration="1m19.724723785s" totalImagesPullingTime="8.811686453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:38.724448472 +0000 UTC m=+80.022790542" watchObservedRunningTime="2026-07-10 18:41:38.724723785 +0000 UTC m=+80.023065853"
Jul 10 18:41:39.719183 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:39.719145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2m65d" event={"ID":"d0e8e075-a99c-4edf-b1ba-3a5504c6734d","Type":"ContainerStarted","Data":"fbc1a24e3f6439f20f0ef7ff1ef3377a0e9a4cb1c4ecba85e3b479c4ee1da736"}
Jul 10 18:41:39.722533 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:39.722504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerStarted","Data":"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"}
Jul 10 18:41:39.722676 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:39.722579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerStarted","Data":"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"}
Jul 10 18:41:39.722676 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:39.722620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerStarted","Data":"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"}
Jul 10 18:41:39.722676 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:39.722662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerStarted","Data":"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"}
Jul 10 18:41:39.722829 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:39.722741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerStarted","Data":"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"}
Jul 10 18:41:39.745494 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:39.745436    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2m65d" podStartSLOduration=72.148057558 podStartE2EDuration="1m20.745418819s" totalImagesPullingTime="8.597361261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:39.742642272 +0000 UTC m=+81.040984336" watchObservedRunningTime="2026-07-10 18:41:39.745418819 +0000 UTC m=+81.043760888"
Jul 10 18:41:39.789811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:39.789744    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.406123245 podStartE2EDuration="20.789717835s" totalImagesPullingTime="17.361919456s" totalInitContainerRuntime="21.675134ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:39.789399058 +0000 UTC m=+81.087741127" watchObservedRunningTime="2026-07-10 18:41:39.789717835 +0000 UTC m=+81.088059904"
Jul 10 18:41:40.170577 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:40.170543    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:40.262512 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:40.262475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:41:40.296868 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:40.296843    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:41.604030 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:41.603992    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:41.721190 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:41.721153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:45.908692 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:45.908659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:45.986573 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:45.986540    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:48.361078 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:48.361032    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:48.461460 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:48.461389    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:41:57.460866 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:57.460834    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bhp9q_7d68bbe0-dc50-45bc-9176-2603887ce446/kube-state-metrics/0.log"
Jul 10 18:41:57.658651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:57.658621    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bhp9q_7d68bbe0-dc50-45bc-9176-2603887ce446/kube-rbac-proxy-main/0.log"
Jul 10 18:41:57.854982 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:57.854889    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bhp9q_7d68bbe0-dc50-45bc-9176-2603887ce446/kube-rbac-proxy-self/0.log"
Jul 10 18:41:58.257663 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:58.257636    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-x96hd_223dc601-b63a-4b63-ae11-091f6ae6f6a7/monitoring-plugin/0.log"
Jul 10 18:41:59.652705 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:59.652678    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvdjd_eacdb005-b9a5-41af-a3a5-25df81aa41a3/init-textfile/0.log"
Jul 10 18:41:59.853573 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:41:59.853545    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvdjd_eacdb005-b9a5-41af-a3a5-25df81aa41a3/node-exporter/0.log"
Jul 10 18:42:00.052976 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:00.052938    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvdjd_eacdb005-b9a5-41af-a3a5-25df81aa41a3/kube-rbac-proxy/0.log"
Jul 10 18:42:00.252363 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:00.252333    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n8xrf_7dccc0b5-9d9e-46c4-8255-5c874c289bfc/kube-rbac-proxy-main/0.log"
Jul 10 18:42:00.453353 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:00.453325    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n8xrf_7dccc0b5-9d9e-46c4-8255-5c874c289bfc/kube-rbac-proxy-self/0.log"
Jul 10 18:42:00.653184 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:00.653146    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n8xrf_7dccc0b5-9d9e-46c4-8255-5c874c289bfc/openshift-state-metrics/0.log"
Jul 10 18:42:00.867928 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:00.867829    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e914c9b-6f51-4175-882e-3b184640fc45/init-config-reloader/0.log"
Jul 10 18:42:01.054808 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:01.054774    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e914c9b-6f51-4175-882e-3b184640fc45/prometheus/0.log"
Jul 10 18:42:01.254367 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:01.254339    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e914c9b-6f51-4175-882e-3b184640fc45/config-reloader/0.log"
Jul 10 18:42:01.453674 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:01.453647    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e914c9b-6f51-4175-882e-3b184640fc45/thanos-sidecar/0.log"
Jul 10 18:42:01.653323 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:01.653252    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e914c9b-6f51-4175-882e-3b184640fc45/kube-rbac-proxy-web/0.log"
Jul 10 18:42:01.853528 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:01.853487    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e914c9b-6f51-4175-882e-3b184640fc45/kube-rbac-proxy/0.log"
Jul 10 18:42:02.053008 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:02.052981    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e914c9b-6f51-4175-882e-3b184640fc45/kube-rbac-proxy-thanos/0.log"
Jul 10 18:42:02.254431 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:02.254406    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-8jxkm_b132a3f7-4f59-4159-9411-8e3562002c89/prometheus-operator/0.log"
Jul 10 18:42:02.453364 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:02.453341    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-8jxkm_b132a3f7-4f59-4159-9411-8e3562002c89/kube-rbac-proxy/0.log"
Jul 10 18:42:02.852337 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:02.852267    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/thanos-query/0.log"
Jul 10 18:42:03.052412 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:03.052382    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/kube-rbac-proxy-web/0.log"
Jul 10 18:42:03.252665 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:03.252636    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/kube-rbac-proxy/0.log"
Jul 10 18:42:03.452697 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:03.452664    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/prom-label-proxy/0.log"
Jul 10 18:42:03.652842 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:03.652765    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/kube-rbac-proxy-rules/0.log"
Jul 10 18:42:03.852662 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:03.852630    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/kube-rbac-proxy-metrics/0.log"
Jul 10 18:42:04.045748 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:04.045716    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:05.053716 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:05.053685    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-x8cj5_a51a24dc-e29e-4dd7-a621-4c169d6d282d/download-server/0.log"
Jul 10 18:42:05.453563 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:05.453502    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nxzsc_fd2847f5-9eae-4319-a9e4-061d9d3f2da1/serve-healthcheck-canary/0.log"
Jul 10 18:42:09.725698 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:09.725656    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vfbvz"
Jul 10 18:42:20.262870 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:20.262827    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:20.327010 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:20.326982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:20.908044 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:20.908017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:32.871830 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:32.871781    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:33.005359 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:33.005321    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:38.261219 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.261182    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:42:38.261632 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.261507    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="prometheus" containerID="cri-o://e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b" gracePeriod=600
Jul 10 18:42:38.261632 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.261522    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy" containerID="cri-o://f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519" gracePeriod=600
Jul 10 18:42:38.261632 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.261561    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy-thanos" containerID="cri-o://eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194" gracePeriod=600
Jul 10 18:42:38.261632 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.261579    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="thanos-sidecar" containerID="cri-o://b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61" gracePeriod=600
Jul 10 18:42:38.261632 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.261588    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="config-reloader" containerID="cri-o://93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e" gracePeriod=600
Jul 10 18:42:38.261851 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.261646    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy-web" containerID="cri-o://afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07" gracePeriod=600
Jul 10 18:42:38.510449 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.510416    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:38.658255 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658159    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-trusted-ca-bundle\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658255 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658207    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-thanos-prometheus-http-client-file\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658255 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658229    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4shxs\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-kube-api-access-4shxs\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658255 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658255    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-config\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658556 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658277    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-serving-certs-ca-bundle\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658556 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658292    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-1\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658556 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658315    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-grpc-tls\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658556 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658335    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-0\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658556 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658366    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-2\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658556 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658400    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-tls-assets\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658556 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658430    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658556 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-kubelet-serving-ca-bundle\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658942 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:38.658942 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658587    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:38.658942 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658780    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:38.658942 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658832    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:38.658942 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658877    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.658942 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658910    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-metrics-client-certs\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.658947    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-metrics-client-ca\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-web-config\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659034    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-config-out\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659056    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-db\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-tls\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659117    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-kube-rbac-proxy\") pod \"6e914c9b-6f51-4175-882e-3b184640fc45\" (UID: \"6e914c9b-6f51-4175-882e-3b184640fc45\") "
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659398    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659418    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659433    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.659445 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.659448    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.660861 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.660069    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:38.660861 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.660725    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:42:38.661883 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.661720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-config" (OuterVolumeSpecName: "config") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.662019 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.661931    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:38.662110 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.662076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:38.662291 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.662204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.662610 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.662515    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:42:38.662610 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.662593    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.662771 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.662671    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.662771 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.662740    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.663944 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.663920    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.664098 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.663973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-kube-api-access-4shxs" (OuterVolumeSpecName: "kube-api-access-4shxs") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "kube-api-access-4shxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:42:38.664290 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.664262    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.664388 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.664367    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-config-out" (OuterVolumeSpecName: "config-out") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:42:38.664559 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.664536    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.674107 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.674083    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-web-config" (OuterVolumeSpecName: "web-config") pod "6e914c9b-6f51-4175-882e-3b184640fc45" (UID: "6e914c9b-6f51-4175-882e-3b184640fc45"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:38.760276 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760237    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760276 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760272    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-tls-assets\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760276 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760285    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760299    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760312    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760324    2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-metrics-client-certs\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760335    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e914c9b-6f51-4175-882e-3b184640fc45-configmap-metrics-client-ca\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760349    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-web-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760361    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-config-out\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760372    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e914c9b-6f51-4175-882e-3b184640fc45-prometheus-k8s-db\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760384    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760395    2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-kube-rbac-proxy\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760407    2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760419    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4shxs\" (UniqueName: \"kubernetes.io/projected/6e914c9b-6f51-4175-882e-3b184640fc45-kube-api-access-4shxs\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760431    2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.760532 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.760442    2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e914c9b-6f51-4175-882e-3b184640fc45-secret-grpc-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:42:38.970612 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970573    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e914c9b-6f51-4175-882e-3b184640fc45" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194" exitCode=0
Jul 10 18:42:38.970612 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970605    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e914c9b-6f51-4175-882e-3b184640fc45" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519" exitCode=0
Jul 10 18:42:38.970612 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970611    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e914c9b-6f51-4175-882e-3b184640fc45" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07" exitCode=0
Jul 10 18:42:38.970612 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970618    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e914c9b-6f51-4175-882e-3b184640fc45" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61" exitCode=0
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970623    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e914c9b-6f51-4175-882e-3b184640fc45" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e" exitCode=0
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970629    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e914c9b-6f51-4175-882e-3b184640fc45" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b" exitCode=0
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970649    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerDied","Data":"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"}
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerDied","Data":"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"}
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerDied","Data":"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"}
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970799    2565 scope.go:117] "RemoveContainer" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerDied","Data":"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"}
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerDied","Data":"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"}
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerDied","Data":"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"}
Jul 10 18:42:38.970968 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.970945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e914c9b-6f51-4175-882e-3b184640fc45","Type":"ContainerDied","Data":"c2a091bb6c37d3293e15cabdfbb688bcb23a56f2de9450152b758e8b41c99376"}
Jul 10 18:42:38.978982 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.978947    2565 scope.go:117] "RemoveContainer" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"
Jul 10 18:42:38.988274 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.988249    2565 scope.go:117] "RemoveContainer" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"
Jul 10 18:42:38.995868 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.995838    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:42:38.996945 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:38.996920    2565 scope.go:117] "RemoveContainer" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"
Jul 10 18:42:39.000557 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.000536    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:42:39.004278 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.004255    2565 scope.go:117] "RemoveContainer" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"
Jul 10 18:42:39.010966 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.010928    2565 scope.go:117] "RemoveContainer" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"
Jul 10 18:42:39.018518 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.018502    2565 scope.go:117] "RemoveContainer" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"
Jul 10 18:42:39.025678 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.025653    2565 scope.go:117] "RemoveContainer" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"
Jul 10 18:42:39.025820 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.025723    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:42:39.025976 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:42:39.025929    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": container with ID starting with eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194 not found: ID does not exist" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"
Jul 10 18:42:39.026050 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.025989    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"} err="failed to get container status \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": rpc error: code = NotFound desc = could not find container \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": container with ID starting with eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194 not found: ID does not exist"
Jul 10 18:42:39.026050 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026038    2565 scope.go:117] "RemoveContainer" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026049    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="prometheus"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026061    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="prometheus"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026074    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="init-config-reloader"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026082    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="init-config-reloader"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026097    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy-web"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026106    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy-web"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026113    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026118    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026130    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="thanos-sidecar"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026135    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="thanos-sidecar"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026141    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy-thanos"
Jul 10 18:42:39.026148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026146    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy-thanos"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026156    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="config-reloader"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026165    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="config-reloader"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026224    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy-web"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026236    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026243    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="prometheus"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026248    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="thanos-sidecar"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026255    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="config-reloader"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026260    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" containerName="kube-rbac-proxy-thanos"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:42:39.026310    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": container with ID starting with f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519 not found: ID does not exist" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026331    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"} err="failed to get container status \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": rpc error: code = NotFound desc = could not find container \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": container with ID starting with f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519 not found: ID does not exist"
Jul 10 18:42:39.026520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026348    2565 scope.go:117] "RemoveContainer" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"
Jul 10 18:42:39.026890 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:42:39.026550    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": container with ID starting with afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07 not found: ID does not exist" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"
Jul 10 18:42:39.026890 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026578    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"} err="failed to get container status \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": rpc error: code = NotFound desc = could not find container \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": container with ID starting with afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07 not found: ID does not exist"
Jul 10 18:42:39.026890 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026601    2565 scope.go:117] "RemoveContainer" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"
Jul 10 18:42:39.026890 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:42:39.026818    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": container with ID starting with b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61 not found: ID does not exist" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"
Jul 10 18:42:39.026890 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026843    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"} err="failed to get container status \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": rpc error: code = NotFound desc = could not find container \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": container with ID starting with b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61 not found: ID does not exist"
Jul 10 18:42:39.026890 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.026862    2565 scope.go:117] "RemoveContainer" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"
Jul 10 18:42:39.027146 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:42:39.027125    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": container with ID starting with 93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e not found: ID does not exist" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"
Jul 10 18:42:39.027179 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.027147    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"} err="failed to get container status \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": rpc error: code = NotFound desc = could not find container \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": container with ID starting with 93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e not found: ID does not exist"
Jul 10 18:42:39.027179 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.027161    2565 scope.go:117] "RemoveContainer" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"
Jul 10 18:42:39.027404 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:42:39.027386    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": container with ID starting with e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b not found: ID does not exist" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"
Jul 10 18:42:39.027450 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.027408    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"} err="failed to get container status \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": rpc error: code = NotFound desc = could not find container \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": container with ID starting with e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b not found: ID does not exist"
Jul 10 18:42:39.027450 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.027422    2565 scope.go:117] "RemoveContainer" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"
Jul 10 18:42:39.027669 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:42:39.027652    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": container with ID starting with ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7 not found: ID does not exist" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"
Jul 10 18:42:39.027712 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.027675    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"} err="failed to get container status \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": rpc error: code = NotFound desc = could not find container \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": container with ID starting with ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7 not found: ID does not exist"
Jul 10 18:42:39.027712 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.027693    2565 scope.go:117] "RemoveContainer" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"
Jul 10 18:42:39.027941 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.027910    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"} err="failed to get container status \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": rpc error: code = NotFound desc = could not find container \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": container with ID starting with eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194 not found: ID does not exist"
Jul 10 18:42:39.027941 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.027939    2565 scope.go:117] "RemoveContainer" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"
Jul 10 18:42:39.028469 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.028444    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"} err="failed to get container status \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": rpc error: code = NotFound desc = could not find container \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": container with ID starting with f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519 not found: ID does not exist"
Jul 10 18:42:39.028593 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.028573    2565 scope.go:117] "RemoveContainer" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"
Jul 10 18:42:39.028843 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.028825    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"} err="failed to get container status \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": rpc error: code = NotFound desc = could not find container \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": container with ID starting with afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07 not found: ID does not exist"
Jul 10 18:42:39.028896 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.028844    2565 scope.go:117] "RemoveContainer" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"
Jul 10 18:42:39.029104 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.029079    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"} err="failed to get container status \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": rpc error: code = NotFound desc = could not find container \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": container with ID starting with b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61 not found: ID does not exist"
Jul 10 18:42:39.029104 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.029103    2565 scope.go:117] "RemoveContainer" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"
Jul 10 18:42:39.029355 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.029333    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"} err="failed to get container status \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": rpc error: code = NotFound desc = could not find container \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": container with ID starting with 93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e not found: ID does not exist"
Jul 10 18:42:39.029425 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.029357    2565 scope.go:117] "RemoveContainer" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"
Jul 10 18:42:39.029700 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.029677    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"} err="failed to get container status \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": rpc error: code = NotFound desc = could not find container \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": container with ID starting with e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b not found: ID does not exist"
Jul 10 18:42:39.029759 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.029702    2565 scope.go:117] "RemoveContainer" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"
Jul 10 18:42:39.029989 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.029966    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"} err="failed to get container status \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": rpc error: code = NotFound desc = could not find container \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": container with ID starting with ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7 not found: ID does not exist"
Jul 10 18:42:39.030038 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.029990    2565 scope.go:117] "RemoveContainer" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"
Jul 10 18:42:39.030279 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.030247    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"} err="failed to get container status \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": rpc error: code = NotFound desc = could not find container \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": container with ID starting with eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194 not found: ID does not exist"
Jul 10 18:42:39.030279 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.030278    2565 scope.go:117] "RemoveContainer" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"
Jul 10 18:42:39.030399 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.030370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.030883 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.030580    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"} err="failed to get container status \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": rpc error: code = NotFound desc = could not find container \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": container with ID starting with f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519 not found: ID does not exist"
Jul 10 18:42:39.030883 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.030603    2565 scope.go:117] "RemoveContainer" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"
Jul 10 18:42:39.030883 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.030833    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"} err="failed to get container status \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": rpc error: code = NotFound desc = could not find container \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": container with ID starting with afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07 not found: ID does not exist"
Jul 10 18:42:39.030883 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.030857    2565 scope.go:117] "RemoveContainer" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"
Jul 10 18:42:39.031164 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.031129    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"} err="failed to get container status \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": rpc error: code = NotFound desc = could not find container \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": container with ID starting with b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61 not found: ID does not exist"
Jul 10 18:42:39.031202 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.031184    2565 scope.go:117] "RemoveContainer" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"
Jul 10 18:42:39.031415 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.031397    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"} err="failed to get container status \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": rpc error: code = NotFound desc = could not find container \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": container with ID starting with 93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e not found: ID does not exist"
Jul 10 18:42:39.031471 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.031417    2565 scope.go:117] "RemoveContainer" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"
Jul 10 18:42:39.031667 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.031638    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"} err="failed to get container status \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": rpc error: code = NotFound desc = could not find container \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": container with ID starting with e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b not found: ID does not exist"
Jul 10 18:42:39.031733 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.031670    2565 scope.go:117] "RemoveContainer" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"
Jul 10 18:42:39.031997 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.031970    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"} err="failed to get container status \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": rpc error: code = NotFound desc = could not find container \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": container with ID starting with ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7 not found: ID does not exist"
Jul 10 18:42:39.031997 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.031996    2565 scope.go:117] "RemoveContainer" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"
Jul 10 18:42:39.032246 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.032228    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"} err="failed to get container status \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": rpc error: code = NotFound desc = could not find container \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": container with ID starting with eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194 not found: ID does not exist"
Jul 10 18:42:39.032318 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.032248    2565 scope.go:117] "RemoveContainer" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"
Jul 10 18:42:39.032462 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.032446    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"} err="failed to get container status \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": rpc error: code = NotFound desc = could not find container \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": container with ID starting with f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519 not found: ID does not exist"
Jul 10 18:42:39.032531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.032463    2565 scope.go:117] "RemoveContainer" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"
Jul 10 18:42:39.032727 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.032682    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"} err="failed to get container status \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": rpc error: code = NotFound desc = could not find container \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": container with ID starting with afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07 not found: ID does not exist"
Jul 10 18:42:39.032817 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.032729    2565 scope.go:117] "RemoveContainer" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"
Jul 10 18:42:39.033060 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033025    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"} err="failed to get container status \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": rpc error: code = NotFound desc = could not find container \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": container with ID starting with b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61 not found: ID does not exist"
Jul 10 18:42:39.033060 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033044    2565 scope.go:117] "RemoveContainer" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"
Jul 10 18:42:39.033538 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033245    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"} err="failed to get container status \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": rpc error: code = NotFound desc = could not find container \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": container with ID starting with 93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e not found: ID does not exist"
Jul 10 18:42:39.033538 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033270    2565 scope.go:117] "RemoveContainer" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"
Jul 10 18:42:39.033538 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033366    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 10 18:42:39.033538 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033411    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ci3nnqgglsdsm\""
Jul 10 18:42:39.033538 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033438    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wdln6\""
Jul 10 18:42:39.033538 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033508    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 10 18:42:39.033538 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033532    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"} err="failed to get container status \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": rpc error: code = NotFound desc = could not find container \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": container with ID starting with e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b not found: ID does not exist"
Jul 10 18:42:39.033923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033555    2565 scope.go:117] "RemoveContainer" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"
Jul 10 18:42:39.033923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033581    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 10 18:42:39.033923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033817    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 10 18:42:39.033923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033824    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"} err="failed to get container status \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": rpc error: code = NotFound desc = could not find container \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": container with ID starting with ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7 not found: ID does not exist"
Jul 10 18:42:39.033923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033847    2565 scope.go:117] "RemoveContainer" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"
Jul 10 18:42:39.034194 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.033947    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 10 18:42:39.034194 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034023    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 10 18:42:39.034293 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034183    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"} err="failed to get container status \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": rpc error: code = NotFound desc = could not find container \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": container with ID starting with eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194 not found: ID does not exist"
Jul 10 18:42:39.034293 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034208    2565 scope.go:117] "RemoveContainer" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"
Jul 10 18:42:39.034463 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034442    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"} err="failed to get container status \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": rpc error: code = NotFound desc = could not find container \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": container with ID starting with f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519 not found: ID does not exist"
Jul 10 18:42:39.034552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034463    2565 scope.go:117] "RemoveContainer" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"
Jul 10 18:42:39.034552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034544    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 10 18:42:39.034709 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034688    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"} err="failed to get container status \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": rpc error: code = NotFound desc = could not find container \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": container with ID starting with afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07 not found: ID does not exist"
Jul 10 18:42:39.034757 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034711    2565 scope.go:117] "RemoveContainer" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"
Jul 10 18:42:39.034935 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034916    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"} err="failed to get container status \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": rpc error: code = NotFound desc = could not find container \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": container with ID starting with b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61 not found: ID does not exist"
Jul 10 18:42:39.035021 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.034936    2565 scope.go:117] "RemoveContainer" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"
Jul 10 18:42:39.035297 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.035277    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"} err="failed to get container status \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": rpc error: code = NotFound desc = could not find container \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": container with ID starting with 93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e not found: ID does not exist"
Jul 10 18:42:39.035297 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.035298    2565 scope.go:117] "RemoveContainer" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"
Jul 10 18:42:39.035511 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.035495    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 10 18:42:39.035593 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.035569    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"} err="failed to get container status \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": rpc error: code = NotFound desc = could not find container \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": container with ID starting with e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b not found: ID does not exist"
Jul 10 18:42:39.035640 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.035596    2565 scope.go:117] "RemoveContainer" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"
Jul 10 18:42:39.035840 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.035812    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 10 18:42:39.035925 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.035844    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"} err="failed to get container status \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": rpc error: code = NotFound desc = could not find container \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": container with ID starting with ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7 not found: ID does not exist"
Jul 10 18:42:39.035925 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.035864    2565 scope.go:117] "RemoveContainer" containerID="eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"
Jul 10 18:42:39.036214 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036191    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194"} err="failed to get container status \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": rpc error: code = NotFound desc = could not find container \"eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194\": container with ID starting with eea7acc565a86f364e71f344d50146d6e164df96b355fcc275704a3948e45194 not found: ID does not exist"
Jul 10 18:42:39.036292 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036215    2565 scope.go:117] "RemoveContainer" containerID="f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"
Jul 10 18:42:39.036349 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036322    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 10 18:42:39.036349 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036338    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 10 18:42:39.036514 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036485    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519"} err="failed to get container status \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": rpc error: code = NotFound desc = could not find container \"f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519\": container with ID starting with f26c971b3b654a8d472ee564cad71ed37ebc94d6ed4130a89ed6a24223c50519 not found: ID does not exist"
Jul 10 18:42:39.036514 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036511    2565 scope.go:117] "RemoveContainer" containerID="afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"
Jul 10 18:42:39.036688 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036672    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 10 18:42:39.036773 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036687    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 10 18:42:39.036858 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036836    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07"} err="failed to get container status \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": rpc error: code = NotFound desc = could not find container \"afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07\": container with ID starting with afd70cecd0beca6c480106ce5b1a885660be457f2c4c2a524934817a648afa07 not found: ID does not exist"
Jul 10 18:42:39.036915 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.036860    2565 scope.go:117] "RemoveContainer" containerID="b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"
Jul 10 18:42:39.037175 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.037149    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61"} err="failed to get container status \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": rpc error: code = NotFound desc = could not find container \"b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61\": container with ID starting with b4904b391c0f17fc9570f40e58c99b35a9be728d9f1996510dd1878627679b61 not found: ID does not exist"
Jul 10 18:42:39.037256 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.037179    2565 scope.go:117] "RemoveContainer" containerID="93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"
Jul 10 18:42:39.037493 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.037445    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e"} err="failed to get container status \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": rpc error: code = NotFound desc = could not find container \"93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e\": container with ID starting with 93d7426d47be2942044dd60ddce52fcd09cfcd9abb75e67f0032334150ef814e not found: ID does not exist"
Jul 10 18:42:39.037590 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.037495    2565 scope.go:117] "RemoveContainer" containerID="e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"
Jul 10 18:42:39.037874 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.037849    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b"} err="failed to get container status \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": rpc error: code = NotFound desc = could not find container \"e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b\": container with ID starting with e30048d0f3531a7f2dfed934c6e052096106990e97f042cbb62bb2b1df87891b not found: ID does not exist"
Jul 10 18:42:39.037978 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.037875    2565 scope.go:117] "RemoveContainer" containerID="ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"
Jul 10 18:42:39.038195 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.038175    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7"} err="failed to get container status \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": rpc error: code = NotFound desc = could not find container \"ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7\": container with ID starting with ab5694486801e5040435b91d4d64d49c72550855b3d39b62e8969ff4f5c766c7 not found: ID does not exist"
Jul 10 18:42:39.039361 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.039334    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 10 18:42:39.039696 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.039678    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 10 18:42:39.050451 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.050420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:42:39.164309 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164271    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229b05ad-2c70-4050-ac5c-9976a0e27f8b-config-out\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164309 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-config\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbpgz\" (UniqueName: \"kubernetes.io/projected/229b05ad-2c70-4050-ac5c-9976a0e27f8b-kube-api-access-mbpgz\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164574    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229b05ad-2c70-4050-ac5c-9976a0e27f8b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164664    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-web-config\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.164852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.165188 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.164884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266087 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.265983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266087 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266087 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266087 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229b05ad-2c70-4050-ac5c-9976a0e27f8b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266263    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-web-config\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229b05ad-2c70-4050-ac5c-9976a0e27f8b-config-out\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-config\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.266681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbpgz\" (UniqueName: \"kubernetes.io/projected/229b05ad-2c70-4050-ac5c-9976a0e27f8b-kube-api-access-mbpgz\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.267711 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.267711 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.267137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.267711 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.267524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.267893 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.267708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.267893 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.266460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.268810 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.268787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.269477 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.269451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.269582 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.269490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-web-config\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.269644 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.269622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229b05ad-2c70-4050-ac5c-9976a0e27f8b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.269755 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.269733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229b05ad-2c70-4050-ac5c-9976a0e27f8b-config-out\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.270115 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.270019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.271351 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.270378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.271351 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.270682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.272033 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.271941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.272849 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.272534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-config\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.272849 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.272584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.272849 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.272612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.272849 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.272801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/229b05ad-2c70-4050-ac5c-9976a0e27f8b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.273472 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.273452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/229b05ad-2c70-4050-ac5c-9976a0e27f8b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.274724 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.274704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbpgz\" (UniqueName: \"kubernetes.io/projected/229b05ad-2c70-4050-ac5c-9976a0e27f8b-kube-api-access-mbpgz\") pod \"prometheus-k8s-0\" (UID: \"229b05ad-2c70-4050-ac5c-9976a0e27f8b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.276766 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.276744    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e914c9b-6f51-4175-882e-3b184640fc45" path="/var/lib/kubelet/pods/6e914c9b-6f51-4175-882e-3b184640fc45/volumes"
Jul 10 18:42:39.341844 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.341801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:39.486092 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.486054    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 10 18:42:39.491657 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:42:39.491620    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229b05ad_2c70_4050_ac5c_9976a0e27f8b.slice/crio-354fc4fae504d390247102ab276f89e570f5f5ba6d82f9183782e306914a2b62 WatchSource:0}: Error finding container 354fc4fae504d390247102ab276f89e570f5f5ba6d82f9183782e306914a2b62: Status 404 returned error can't find the container with id 354fc4fae504d390247102ab276f89e570f5f5ba6d82f9183782e306914a2b62
Jul 10 18:42:39.981401 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.977431    2565 generic.go:356] "Generic (PLEG): container finished" podID="229b05ad-2c70-4050-ac5c-9976a0e27f8b" containerID="f25469e8051c729b781d37c96d9776a264caedf8f5b9fad8610824825428b003" exitCode=0
Jul 10 18:42:39.981401 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.977495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229b05ad-2c70-4050-ac5c-9976a0e27f8b","Type":"ContainerDied","Data":"f25469e8051c729b781d37c96d9776a264caedf8f5b9fad8610824825428b003"}
Jul 10 18:42:39.981401 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:39.977621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229b05ad-2c70-4050-ac5c-9976a0e27f8b","Type":"ContainerStarted","Data":"354fc4fae504d390247102ab276f89e570f5f5ba6d82f9183782e306914a2b62"}
Jul 10 18:42:40.985459 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:40.985418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229b05ad-2c70-4050-ac5c-9976a0e27f8b","Type":"ContainerStarted","Data":"efc8a78cb57da804a6ad56d8f78b7e21707c5d0230b9b609cf4d603ea868df97"}
Jul 10 18:42:40.986059 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:40.985496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229b05ad-2c70-4050-ac5c-9976a0e27f8b","Type":"ContainerStarted","Data":"809ee3143372f68c8f51fcf440a0d8407e8d5e1e99a1eeb15582bdd58d86e3e8"}
Jul 10 18:42:40.986059 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:40.985535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229b05ad-2c70-4050-ac5c-9976a0e27f8b","Type":"ContainerStarted","Data":"9dc38f6aad2240f16ca1c4da279f9bf3646635899f8ae97897b778b4a9ea0d76"}
Jul 10 18:42:40.986059 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:40.985583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229b05ad-2c70-4050-ac5c-9976a0e27f8b","Type":"ContainerStarted","Data":"ecde4661b8f53c933a4293295d1e3b04f82d5eb837c05eba4f43573d4140c0c5"}
Jul 10 18:42:40.986059 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:40.985625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229b05ad-2c70-4050-ac5c-9976a0e27f8b","Type":"ContainerStarted","Data":"b5f6e08c3c6c03445c286c479bd6f636268accce1b47b56b039174c987ede30a"}
Jul 10 18:42:40.986059 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:40.985660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229b05ad-2c70-4050-ac5c-9976a0e27f8b","Type":"ContainerStarted","Data":"2e748abecc342986914306f7c22c1d2b496428a16821c185b2db66e633317c08"}
Jul 10 18:42:41.014405 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:41.014340    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.99509457 podStartE2EDuration="2.014322785s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.228215ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:42:41.011986073 +0000 UTC m=+142.310328144" watchObservedRunningTime="2026-07-10 18:42:41.014322785 +0000 UTC m=+142.312664853"
Jul 10 18:42:42.470883 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:42.470847    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:42.608209 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:42.608178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:42.670971 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:42.670932    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:43.203797 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:43.203761    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:43.270433 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:43.270393    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:44.342130 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:44.342096    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:42:53.656350 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:53.656315    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:53.756791 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:53.756760    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:42:53.844429 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:42:53.844403    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:05.892017 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:05.891981    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:06.008905 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:06.008872    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:23.001265 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:23.001230    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:23.074296 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:23.074258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:25.243157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:25.243119    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:25.343482 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:25.343447    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:26.247194 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:26.247158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:26.351071 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:26.351038    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:26.408986 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:26.408936    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:39.342095 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:39.342065    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:39.358041 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:39.358014    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:40.238763 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:40.238733    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 10 18:43:45.920417 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:45.920380    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:46.023409 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:46.023370    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:48.147545 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:48.147507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:48.223502 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:48.223472    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:48.874232 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:48.874202    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:48.996123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:48.996096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:49.093230 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:49.093197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:49.484975 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:49.484925    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:49.552129 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:49.552100    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:51.967446 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:51.967413    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:52.033287 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:52.033256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:59.822524 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:59.822485    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:59.878174 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:59.878127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:43:59.939256 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:43:59.939209    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:00.900947 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:44:00.900912    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:01.001500 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:44:01.001472    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:59.837705 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:44:59.837629    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:44:59.897660 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:44:59.897629    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:02.490570 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:02.490533    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:02.591278 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:02.591238    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:19.104462 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:19.104430    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:45:19.109019 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:19.108998    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 18:45:19.109230 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:19.109213    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:45:29.678755 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:29.678719    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:30.929319 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:30.929280    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:54.334620 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:54.334576    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:54.422711 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:54.422678    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:58.393217 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:58.393181    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:58.494546 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:58.494517    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:45:59.858947 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:45:59.858916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:00.187671 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:00.187627    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:00.359569 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:00.359536    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:00.521933 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:00.521848    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:00.625938 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:00.625891    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:00.840553 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:00.840468    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:00.939560 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:00.939510    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:01.299169 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:01.299122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:01.455509 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:01.455452    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:01.573370 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:01.573256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:01.684710 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:01.684677    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:04.764142 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:04.764108    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:52.075121 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:52.075082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:46:52.175763 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:46:52.175722    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:05.768459 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.768410    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:05.863383 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.863350    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:05.881142 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.881113    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qbzzv"]
Jul 10 18:47:05.883486 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.883461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:05.886451 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.886296    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q6nrq\""
Jul 10 18:47:05.886573 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.886558    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 10 18:47:05.887108 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.887090    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 18:47:05.887169 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.887145    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 10 18:47:05.897025 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.897005    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qbzzv"]
Jul 10 18:47:05.960780 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.960736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czl8h\" (UniqueName: \"kubernetes.io/projected/535af7ea-87c5-4685-b6b2-dc12bd2e77f6-kube-api-access-czl8h\") pod \"seaweedfs-64568bcd49-qbzzv\" (UID: \"535af7ea-87c5-4685-b6b2-dc12bd2e77f6\") " pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:05.960780 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.960772    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/535af7ea-87c5-4685-b6b2-dc12bd2e77f6-data\") pod \"seaweedfs-64568bcd49-qbzzv\" (UID: \"535af7ea-87c5-4685-b6b2-dc12bd2e77f6\") " pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:05.961566 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:05.961541    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:06.061860 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.061759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czl8h\" (UniqueName: \"kubernetes.io/projected/535af7ea-87c5-4685-b6b2-dc12bd2e77f6-kube-api-access-czl8h\") pod \"seaweedfs-64568bcd49-qbzzv\" (UID: \"535af7ea-87c5-4685-b6b2-dc12bd2e77f6\") " pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:06.061860 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.061803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/535af7ea-87c5-4685-b6b2-dc12bd2e77f6-data\") pod \"seaweedfs-64568bcd49-qbzzv\" (UID: \"535af7ea-87c5-4685-b6b2-dc12bd2e77f6\") " pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:06.062232 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.062212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/535af7ea-87c5-4685-b6b2-dc12bd2e77f6-data\") pod \"seaweedfs-64568bcd49-qbzzv\" (UID: \"535af7ea-87c5-4685-b6b2-dc12bd2e77f6\") " pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:06.071934 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.071907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czl8h\" (UniqueName: \"kubernetes.io/projected/535af7ea-87c5-4685-b6b2-dc12bd2e77f6-kube-api-access-czl8h\") pod \"seaweedfs-64568bcd49-qbzzv\" (UID: \"535af7ea-87c5-4685-b6b2-dc12bd2e77f6\") " pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:06.072467 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.072436    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:06.192478 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.192445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:06.318483 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.318461    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qbzzv"]
Jul 10 18:47:06.321330 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:47:06.321299    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535af7ea_87c5_4685_b6b2_dc12bd2e77f6.slice/crio-7dc96e0e25b55d06cea15ef323d944d392c6d20898271926c7ca6bab0d95f45f WatchSource:0}: Error finding container 7dc96e0e25b55d06cea15ef323d944d392c6d20898271926c7ca6bab0d95f45f: Status 404 returned error can't find the container with id 7dc96e0e25b55d06cea15ef323d944d392c6d20898271926c7ca6bab0d95f45f
Jul 10 18:47:06.322560 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.322540    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:47:06.619630 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.619528    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:06.675229 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.675195    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:06.984486 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:06.984445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qbzzv" event={"ID":"535af7ea-87c5-4685-b6b2-dc12bd2e77f6","Type":"ContainerStarted","Data":"7dc96e0e25b55d06cea15ef323d944d392c6d20898271926c7ca6bab0d95f45f"}
Jul 10 18:47:09.998302 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:09.998267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qbzzv" event={"ID":"535af7ea-87c5-4685-b6b2-dc12bd2e77f6","Type":"ContainerStarted","Data":"0f19f0a4b1133478d5af049c1b0005ad6443536e5aa7c7ac12aaf7b79e7d9af3"}
Jul 10 18:47:09.998680 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:09.998395    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:10.017511 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:10.017461    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qbzzv" podStartSLOduration=1.722869571 podStartE2EDuration="5.017446111s" totalImagesPullingTime="3.29457654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:10.015735896 +0000 UTC m=+411.314077965" watchObservedRunningTime="2026-07-10 18:47:10.017446111 +0000 UTC m=+411.315788181"
Jul 10 18:47:16.004853 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:16.004825    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qbzzv"
Jul 10 18:47:48.728101 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:48.728020    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:48.829014 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:48.828975    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:59.163047 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:59.163011    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:59.308758 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:59.308719    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:47:59.945159 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:47:59.945123    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:09.868319 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:09.868278    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:09.997259 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:09.997224    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:16.554536 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:16.554494    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:16.655067 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:16.655033    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:16.719973 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:16.719927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:16.815946 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:16.815859    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:17.312874 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:17.312835    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:17.414143 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:17.414112    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:32.592907 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:32.592873    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2kl9d"]
Jul 10 18:48:32.596121 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:32.596104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2kl9d"
Jul 10 18:48:32.604013 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:32.603989    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2kl9d"]
Jul 10 18:48:32.746947 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:32.746905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbsw\" (UniqueName: \"kubernetes.io/projected/3c1cd301-000e-47ff-bf64-4588692e7cf7-kube-api-access-xhbsw\") pod \"s3-init-2kl9d\" (UID: \"3c1cd301-000e-47ff-bf64-4588692e7cf7\") " pod="kserve/s3-init-2kl9d"
Jul 10 18:48:32.847733 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:32.847647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbsw\" (UniqueName: \"kubernetes.io/projected/3c1cd301-000e-47ff-bf64-4588692e7cf7-kube-api-access-xhbsw\") pod \"s3-init-2kl9d\" (UID: \"3c1cd301-000e-47ff-bf64-4588692e7cf7\") " pod="kserve/s3-init-2kl9d"
Jul 10 18:48:32.856967 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:32.856924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbsw\" (UniqueName: \"kubernetes.io/projected/3c1cd301-000e-47ff-bf64-4588692e7cf7-kube-api-access-xhbsw\") pod \"s3-init-2kl9d\" (UID: \"3c1cd301-000e-47ff-bf64-4588692e7cf7\") " pod="kserve/s3-init-2kl9d"
Jul 10 18:48:32.918651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:32.918613    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2kl9d"
Jul 10 18:48:33.047666 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:33.047632    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2kl9d"]
Jul 10 18:48:33.050582 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:48:33.050553    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c1cd301_000e_47ff_bf64_4588692e7cf7.slice/crio-d6780300ed0f17c32b82a7fa783f20bf54bac2407fa8d47888ccf6fc4ec2b855 WatchSource:0}: Error finding container d6780300ed0f17c32b82a7fa783f20bf54bac2407fa8d47888ccf6fc4ec2b855: Status 404 returned error can't find the container with id d6780300ed0f17c32b82a7fa783f20bf54bac2407fa8d47888ccf6fc4ec2b855
Jul 10 18:48:33.308638 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:33.308605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kl9d" event={"ID":"3c1cd301-000e-47ff-bf64-4588692e7cf7","Type":"ContainerStarted","Data":"d6780300ed0f17c32b82a7fa783f20bf54bac2407fa8d47888ccf6fc4ec2b855"}
Jul 10 18:48:38.331698 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:38.331668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kl9d" event={"ID":"3c1cd301-000e-47ff-bf64-4588692e7cf7","Type":"ContainerStarted","Data":"5f779e272bb273d0a20ec72b78909aa9db29961c0aee85cbc8e2803c8aa0cb2b"}
Jul 10 18:48:38.345941 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:38.345888    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2kl9d" podStartSLOduration=1.696115949 podStartE2EDuration="6.345869241s" totalImagesPullingTime="4.649753292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:38.345752608 +0000 UTC m=+499.644094677" watchObservedRunningTime="2026-07-10 18:48:38.345869241 +0000 UTC m=+499.644211311"
Jul 10 18:48:41.344602 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:41.344556    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c1cd301-000e-47ff-bf64-4588692e7cf7" containerID="5f779e272bb273d0a20ec72b78909aa9db29961c0aee85cbc8e2803c8aa0cb2b" exitCode=0
Jul 10 18:48:41.345085 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:41.344630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kl9d" event={"ID":"3c1cd301-000e-47ff-bf64-4588692e7cf7","Type":"ContainerDied","Data":"5f779e272bb273d0a20ec72b78909aa9db29961c0aee85cbc8e2803c8aa0cb2b"}
Jul 10 18:48:42.477587 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:42.477563    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2kl9d"
Jul 10 18:48:42.630236 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:42.630156    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhbsw\" (UniqueName: \"kubernetes.io/projected/3c1cd301-000e-47ff-bf64-4588692e7cf7-kube-api-access-xhbsw\") pod \"3c1cd301-000e-47ff-bf64-4588692e7cf7\" (UID: \"3c1cd301-000e-47ff-bf64-4588692e7cf7\") "
Jul 10 18:48:42.632521 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:42.632491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1cd301-000e-47ff-bf64-4588692e7cf7-kube-api-access-xhbsw" (OuterVolumeSpecName: "kube-api-access-xhbsw") pod "3c1cd301-000e-47ff-bf64-4588692e7cf7" (UID: "3c1cd301-000e-47ff-bf64-4588692e7cf7"). InnerVolumeSpecName "kube-api-access-xhbsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:48:42.731137 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:42.731094    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbsw\" (UniqueName: \"kubernetes.io/projected/3c1cd301-000e-47ff-bf64-4588692e7cf7-kube-api-access-xhbsw\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:48:43.352923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:43.352893    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2kl9d"
Jul 10 18:48:43.352923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:43.352912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kl9d" event={"ID":"3c1cd301-000e-47ff-bf64-4588692e7cf7","Type":"ContainerDied","Data":"d6780300ed0f17c32b82a7fa783f20bf54bac2407fa8d47888ccf6fc4ec2b855"}
Jul 10 18:48:43.353130 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:43.352948    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6780300ed0f17c32b82a7fa783f20bf54bac2407fa8d47888ccf6fc4ec2b855"
Jul 10 18:48:43.876082 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:43.876046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:43.976365 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:43.976330    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:52.614709 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.614675    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:52.658882 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.658849    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"]
Jul 10 18:48:52.659182 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.659168    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1cd301-000e-47ff-bf64-4588692e7cf7" containerName="s3-init"
Jul 10 18:48:52.659232 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.659184    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1cd301-000e-47ff-bf64-4588692e7cf7" containerName="s3-init"
Jul 10 18:48:52.659266 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.659253    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1cd301-000e-47ff-bf64-4588692e7cf7" containerName="s3-init"
Jul 10 18:48:52.662531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.662512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.665966 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.665921    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ba55b-predictor-serving-cert\""
Jul 10 18:48:52.666062 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.666027    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ba55b-kube-rbac-proxy-sar-config\""
Jul 10 18:48:52.666166 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.666130    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:48:52.666206 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.666198    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9w5fm\""
Jul 10 18:48:52.666825 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.666808    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:48:52.675123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.675098    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"]
Jul 10 18:48:52.821767 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.821727    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-success-200-isvc-ba55b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.821946 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.821792    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhs6\" (UniqueName: \"kubernetes.io/projected/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-kube-api-access-kdhs6\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.821946 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.821870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.922630 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.922570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhs6\" (UniqueName: \"kubernetes.io/projected/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-kube-api-access-kdhs6\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.922630 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.922637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.922867 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.922674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-success-200-isvc-ba55b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.922867 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:48:52.922809    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-serving-cert: secret "success-200-isvc-ba55b-predictor-serving-cert" not found
Jul 10 18:48:52.922945 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:48:52.922879    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls podName:62e9f59e-c117-4fb6-bc58-e0569b8f9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:48:53.422857788 +0000 UTC m=+514.721199842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls") pod "success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" (UID: "62e9f59e-c117-4fb6-bc58-e0569b8f9ade") : secret "success-200-isvc-ba55b-predictor-serving-cert" not found
Jul 10 18:48:52.923388 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.923369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-success-200-isvc-ba55b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.929432 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.929400    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:52.935095 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.935076    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhs6\" (UniqueName: \"kubernetes.io/projected/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-kube-api-access-kdhs6\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:52.965735 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.965699    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"]
Jul 10 18:48:52.969208 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.969186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:52.971682 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.971661    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 10 18:48:52.971793 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.971708    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 10 18:48:52.980345 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:52.980321    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"]
Jul 10 18:48:53.023728 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.023700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.023906 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.023745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxq5f\" (UniqueName: \"kubernetes.io/projected/1e44c877-3e7a-488f-9e44-5277532972c7-kube-api-access-pxq5f\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.023906 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.023765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e44c877-3e7a-488f-9e44-5277532972c7-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.023906 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.023845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e44c877-3e7a-488f-9e44-5277532972c7-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.103396 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.103364    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:53.124558 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.124523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.124689 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.124571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxq5f\" (UniqueName: \"kubernetes.io/projected/1e44c877-3e7a-488f-9e44-5277532972c7-kube-api-access-pxq5f\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.124689 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:48:53.124662    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 10 18:48:53.124689 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.124678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e44c877-3e7a-488f-9e44-5277532972c7-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.124812 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:48:53.124719    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls podName:1e44c877-3e7a-488f-9e44-5277532972c7 nodeName:}" failed. No retries permitted until 2026-07-10 18:48:53.624704241 +0000 UTC m=+514.923046292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls") pod "isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" (UID: "1e44c877-3e7a-488f-9e44-5277532972c7") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 10 18:48:53.124812 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.124741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e44c877-3e7a-488f-9e44-5277532972c7-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.125144 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.125128    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e44c877-3e7a-488f-9e44-5277532972c7-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.125369 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.125348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e44c877-3e7a-488f-9e44-5277532972c7-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.134555 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.134533    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxq5f\" (UniqueName: \"kubernetes.io/projected/1e44c877-3e7a-488f-9e44-5277532972c7-kube-api-access-pxq5f\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.157346 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.157313    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"]
Jul 10 18:48:53.160698 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.160682    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.163109 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.163085    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ba55b-predictor-serving-cert\""
Jul 10 18:48:53.163236 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.163130    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ba55b-kube-rbac-proxy-sar-config\""
Jul 10 18:48:53.171499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.171477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"]
Jul 10 18:48:53.203036 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.202940    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:53.225072 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.225043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/a66ce281-c2d2-41c1-803c-303353635f5d-kube-api-access-wckt9\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.225250 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.225092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66ce281-c2d2-41c1-803c-303353635f5d-error-404-isvc-ba55b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.225250 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.225171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66ce281-c2d2-41c1-803c-303353635f5d-proxy-tls\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.302221 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.302192    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:53.325510 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.325464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66ce281-c2d2-41c1-803c-303353635f5d-proxy-tls\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.325704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.325519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/a66ce281-c2d2-41c1-803c-303353635f5d-kube-api-access-wckt9\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.325704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.325573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66ce281-c2d2-41c1-803c-303353635f5d-error-404-isvc-ba55b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.326302 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.326280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66ce281-c2d2-41c1-803c-303353635f5d-error-404-isvc-ba55b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.328147 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.328126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66ce281-c2d2-41c1-803c-303353635f5d-proxy-tls\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.338521 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.338477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/a66ce281-c2d2-41c1-803c-303353635f5d-kube-api-access-wckt9\") pod \"error-404-isvc-ba55b-predictor-578475f44-lgsng\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") " pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.398486 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.398446    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:53.426641 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.426601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:53.429135 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.429118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls\") pod \"success-200-isvc-ba55b-predictor-575d4874b7-tx7wh\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") " pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:53.470970 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.470870    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:48:53.498400 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.498368    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:53.573543 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.573492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:48:53.598593 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.598564    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:53.607630 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.607609    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"]
Jul 10 18:48:53.610272 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:48:53.610247    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66ce281_c2d2_41c1_803c_303353635f5d.slice/crio-6cc55df1b3df63d6107b988555dba303a6751587f6fed88db9292a572356d230 WatchSource:0}: Error finding container 6cc55df1b3df63d6107b988555dba303a6751587f6fed88db9292a572356d230: Status 404 returned error can't find the container with id 6cc55df1b3df63d6107b988555dba303a6751587f6fed88db9292a572356d230
Jul 10 18:48:53.628642 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.628619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.632329 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.632269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.708472 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.708444    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"]
Jul 10 18:48:53.710550 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:48:53.710527    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e9f59e_c117_4fb6_bc58_e0569b8f9ade.slice/crio-4630f02a1c324058a886d2f5a0dd5759a4555b9b90ef068044476cb544800639 WatchSource:0}: Error finding container 4630f02a1c324058a886d2f5a0dd5759a4555b9b90ef068044476cb544800639: Status 404 returned error can't find the container with id 4630f02a1c324058a886d2f5a0dd5759a4555b9b90ef068044476cb544800639
Jul 10 18:48:53.781266 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.781138    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:48:53.865544 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.865500    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"]
Jul 10 18:48:53.871298 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.871272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:53.873694 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.873667    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 10 18:48:53.873814 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.873763    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 10 18:48:53.878852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.878826    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"]
Jul 10 18:48:53.879605 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.879585    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:48:53.931560 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.931528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jnvv\" (UniqueName: \"kubernetes.io/projected/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kube-api-access-8jnvv\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:53.931681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.931584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:53.931681 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.931640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:53.931811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:53.931729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.018811 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.018780    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"]
Jul 10 18:48:54.022321 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:48:54.022289    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e44c877_3e7a_488f_9e44_5277532972c7.slice/crio-89edcd6349e77fd9897ea447b6a92f17b1c8f8cb9b4e86efc098f65c06490d8b WatchSource:0}: Error finding container 89edcd6349e77fd9897ea447b6a92f17b1c8f8cb9b4e86efc098f65c06490d8b: Status 404 returned error can't find the container with id 89edcd6349e77fd9897ea447b6a92f17b1c8f8cb9b4e86efc098f65c06490d8b
Jul 10 18:48:54.032812 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.032753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.032901 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.032813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jnvv\" (UniqueName: \"kubernetes.io/projected/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kube-api-access-8jnvv\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.032978 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.032913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.032978 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.032946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.033265 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.033243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.033421 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.033404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.035353 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.035337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.042328 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.042306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jnvv\" (UniqueName: \"kubernetes.io/projected/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kube-api-access-8jnvv\") pod \"isvc-sklearn-graph-2-predictor-745b467f-g9mb6\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.182891 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.182851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:48:54.346712 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.346573    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"]
Jul 10 18:48:54.351495 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:48:54.351448    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd7ff70_45d4_4a6e_a2c7_094d6ef757c2.slice/crio-f4ab4f2cfa399d2de28724363190e9f239a1112322d8d1e183c949d5853e1b98 WatchSource:0}: Error finding container f4ab4f2cfa399d2de28724363190e9f239a1112322d8d1e183c949d5853e1b98: Status 404 returned error can't find the container with id f4ab4f2cfa399d2de28724363190e9f239a1112322d8d1e183c949d5853e1b98
Jul 10 18:48:54.402166 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.401246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" event={"ID":"1e44c877-3e7a-488f-9e44-5277532972c7","Type":"ContainerStarted","Data":"89edcd6349e77fd9897ea447b6a92f17b1c8f8cb9b4e86efc098f65c06490d8b"}
Jul 10 18:48:54.415560 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.415519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" event={"ID":"62e9f59e-c117-4fb6-bc58-e0569b8f9ade","Type":"ContainerStarted","Data":"4630f02a1c324058a886d2f5a0dd5759a4555b9b90ef068044476cb544800639"}
Jul 10 18:48:54.419500 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.419370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" event={"ID":"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2","Type":"ContainerStarted","Data":"f4ab4f2cfa399d2de28724363190e9f239a1112322d8d1e183c949d5853e1b98"}
Jul 10 18:48:54.421599 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:48:54.421518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" event={"ID":"a66ce281-c2d2-41c1-803c-303353635f5d","Type":"ContainerStarted","Data":"6cc55df1b3df63d6107b988555dba303a6751587f6fed88db9292a572356d230"}
Jul 10 18:49:12.582838 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:12.582742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" event={"ID":"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2","Type":"ContainerStarted","Data":"c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198"}
Jul 10 18:49:12.584667 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:12.584640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" event={"ID":"a66ce281-c2d2-41c1-803c-303353635f5d","Type":"ContainerStarted","Data":"c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480"}
Jul 10 18:49:12.586362 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:12.586326    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" event={"ID":"1e44c877-3e7a-488f-9e44-5277532972c7","Type":"ContainerStarted","Data":"86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09"}
Jul 10 18:49:12.588088 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:12.588062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" event={"ID":"62e9f59e-c117-4fb6-bc58-e0569b8f9ade","Type":"ContainerStarted","Data":"57c705f22496bc9fb9b92c3d3022c213a5461d288342cc2634338d83a7ef2adc"}
Jul 10 18:49:15.606399 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:15.606351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" event={"ID":"a66ce281-c2d2-41c1-803c-303353635f5d","Type":"ContainerStarted","Data":"a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1"}
Jul 10 18:49:15.606851 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:15.606541    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:49:15.606851 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:15.606600    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:49:15.608141 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:15.608113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" event={"ID":"62e9f59e-c117-4fb6-bc58-e0569b8f9ade","Type":"ContainerStarted","Data":"cebe2a0cc44ae8ec54e7cae52a339d2f7b5e03d049a1b1ed61595854642f0d42"}
Jul 10 18:49:15.608272 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:15.608215    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 18:49:15.608322 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:15.608303    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:49:15.624809 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:15.624764    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podStartSLOduration=1.280561549 podStartE2EDuration="22.624745982s" totalImagesPullingTime="21.344184433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:15.624104357 +0000 UTC m=+536.922446425" watchObservedRunningTime="2026-07-10 18:49:15.624745982 +0000 UTC m=+536.923088051"
Jul 10 18:49:15.642307 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:15.642264    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podStartSLOduration=2.377257382 podStartE2EDuration="23.642249819s" totalImagesPullingTime="21.264992437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:15.641608029 +0000 UTC m=+536.939950111" watchObservedRunningTime="2026-07-10 18:49:15.642249819 +0000 UTC m=+536.940591888"
Jul 10 18:49:16.614247 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:16.614212    2565 generic.go:356] "Generic (PLEG): container finished" podID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerID="c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198" exitCode=0
Jul 10 18:49:16.614661 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:16.614283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" event={"ID":"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2","Type":"ContainerDied","Data":"c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198"}
Jul 10 18:49:16.616019 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:16.616000    2565 generic.go:356] "Generic (PLEG): container finished" podID="1e44c877-3e7a-488f-9e44-5277532972c7" containerID="86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09" exitCode=0
Jul 10 18:49:16.616102 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:16.616081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" event={"ID":"1e44c877-3e7a-488f-9e44-5277532972c7","Type":"ContainerDied","Data":"86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09"}
Jul 10 18:49:16.616474 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:16.616452    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 18:49:16.616627 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:16.616460    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:49:16.617772 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:16.617738    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 18:49:17.622597 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:17.622553    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 18:49:21.621595 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:21.621561    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:49:21.622126 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:21.622098    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 18:49:22.627479 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:22.627452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:49:22.628042 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:22.628011    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 18:49:23.651594 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:23.651545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" event={"ID":"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2","Type":"ContainerStarted","Data":"2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0"}
Jul 10 18:49:23.652157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:23.651639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" event={"ID":"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2","Type":"ContainerStarted","Data":"502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151"}
Jul 10 18:49:23.652157 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:23.651889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:49:23.654568 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:23.654531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" event={"ID":"1e44c877-3e7a-488f-9e44-5277532972c7","Type":"ContainerStarted","Data":"016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae"}
Jul 10 18:49:23.654716 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:23.654594    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" event={"ID":"1e44c877-3e7a-488f-9e44-5277532972c7","Type":"ContainerStarted","Data":"4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab"}
Jul 10 18:49:23.654804 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:23.654787    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:49:23.671839 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:23.671781    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podStartSLOduration=3.930347894 podStartE2EDuration="30.671761001s" totalImagesPullingTime="22.808075274s" totalInitContainerRuntime="3.933337833s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:23.670473743 +0000 UTC m=+544.968815813" watchObservedRunningTime="2026-07-10 18:49:23.671761001 +0000 UTC m=+544.970103071"
Jul 10 18:49:23.688801 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:23.688752    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podStartSLOduration=7.876812371 podStartE2EDuration="31.68873495s" totalImagesPullingTime="20.171635815s" totalInitContainerRuntime="3.640286764s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:23.687649716 +0000 UTC m=+544.985991796" watchObservedRunningTime="2026-07-10 18:49:23.68873495 +0000 UTC m=+544.987077019"
Jul 10 18:49:24.659888 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:24.659856    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:49:24.660346 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:24.659915    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:49:24.660852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:24.660826    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 18:49:24.660930 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:24.660904    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:49:25.663607 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:25.663570    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:49:25.664151 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:25.663561    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 18:49:30.667647 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:30.667618    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:49:30.668101 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:30.668040    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:49:30.668101 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:30.668072    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 18:49:30.668501 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:30.668478    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:49:31.622288 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:31.622247    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 18:49:32.628196 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:32.628152    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 18:49:40.668103 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:40.668034    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 18:49:40.668611 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:40.668470    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:49:41.622848 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:41.622806    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 18:49:42.627944 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:42.627908    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 18:49:50.668354 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:50.668305    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 18:49:50.668842 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:50.668458    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:49:51.623042 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:51.623001    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 18:49:52.628331 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:49:52.628291    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 18:50:00.668095 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:00.668048    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 18:50:00.668612 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:00.668501    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:50:01.623048 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:01.623005    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 18:50:02.629113 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:02.629082    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:50:10.669278 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:10.669225    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:50:10.669278 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:10.669227    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 18:50:11.623134 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:11.623105    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:50:18.749971 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:18.749927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:18.850837 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:18.850805    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:19.131615 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:19.131531    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:50:19.135145 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:19.135119    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:50:20.668699 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:20.668660    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 18:50:20.669092 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:20.668660    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:50:22.723001 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.722947    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"]
Jul 10 18:50:22.726788 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.726763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:22.726917 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.726805    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:22.729253 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.729233    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ba55b-kube-rbac-proxy-sar-config\""
Jul 10 18:50:22.729570 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.729548    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ba55b-serving-cert\""
Jul 10 18:50:22.738692 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.738668    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"]
Jul 10 18:50:22.816340 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.816296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adcdc57d-f1b6-4e05-a1bc-ec031351202e-openshift-service-ca-bundle\") pod \"switch-graph-ba55b-7558d56dd-crnz5\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") " pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:22.816340 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.816346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls\") pod \"switch-graph-ba55b-7558d56dd-crnz5\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") " pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:22.917352 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.917318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls\") pod \"switch-graph-ba55b-7558d56dd-crnz5\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") " pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:22.917517 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.917411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adcdc57d-f1b6-4e05-a1bc-ec031351202e-openshift-service-ca-bundle\") pod \"switch-graph-ba55b-7558d56dd-crnz5\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") " pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:22.917517 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:50:22.917463    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-ba55b-serving-cert: secret "switch-graph-ba55b-serving-cert" not found
Jul 10 18:50:22.917586 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:50:22.917526    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls podName:adcdc57d-f1b6-4e05-a1bc-ec031351202e nodeName:}" failed. No retries permitted until 2026-07-10 18:50:23.417508556 +0000 UTC m=+604.715850603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls") pod "switch-graph-ba55b-7558d56dd-crnz5" (UID: "adcdc57d-f1b6-4e05-a1bc-ec031351202e") : secret "switch-graph-ba55b-serving-cert" not found
Jul 10 18:50:22.918065 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:22.918049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adcdc57d-f1b6-4e05-a1bc-ec031351202e-openshift-service-ca-bundle\") pod \"switch-graph-ba55b-7558d56dd-crnz5\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") " pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:23.121670 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:23.121594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:23.422277 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:23.422244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls\") pod \"switch-graph-ba55b-7558d56dd-crnz5\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") " pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:23.424871 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:23.424846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls\") pod \"switch-graph-ba55b-7558d56dd-crnz5\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") " pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:23.637755 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:23.637715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:23.772774 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:23.772738    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"]
Jul 10 18:50:23.775787 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:50:23.775758    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcdc57d_f1b6_4e05_a1bc_ec031351202e.slice/crio-4ad15db448321085140b7f70cb53df3e4f7e460bdf776ec9bc607795a479726f WatchSource:0}: Error finding container 4ad15db448321085140b7f70cb53df3e4f7e460bdf776ec9bc607795a479726f: Status 404 returned error can't find the container with id 4ad15db448321085140b7f70cb53df3e4f7e460bdf776ec9bc607795a479726f
Jul 10 18:50:23.911400 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:23.911367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" event={"ID":"adcdc57d-f1b6-4e05-a1bc-ec031351202e","Type":"ContainerStarted","Data":"4ad15db448321085140b7f70cb53df3e4f7e460bdf776ec9bc607795a479726f"}
Jul 10 18:50:26.929577 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:26.929541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" event={"ID":"adcdc57d-f1b6-4e05-a1bc-ec031351202e","Type":"ContainerStarted","Data":"a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1"}
Jul 10 18:50:26.930033 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:26.929700    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:26.949473 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:26.949417    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" podStartSLOduration=2.757968385 podStartE2EDuration="4.949400195s" totalImagesPullingTime="2.19143181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:50:26.949174936 +0000 UTC m=+608.247517032" watchObservedRunningTime="2026-07-10 18:50:26.949400195 +0000 UTC m=+608.247742267"
Jul 10 18:50:30.668704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:30.668667    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:50:30.671039 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:30.669115    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:50:32.941383 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:32.941351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:35.273042 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:35.273017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:50:36.836543 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:36.836501    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"]
Jul 10 18:50:36.836937 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:36.836716    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b" containerID="cri-o://a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1" gracePeriod=30
Jul 10 18:50:36.864277 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:36.864249    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:37.054276 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.054245    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"]
Jul 10 18:50:37.058030 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.058005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.060773 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.060741    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d6b08-kube-rbac-proxy-sar-config\""
Jul 10 18:50:37.060889 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.060790    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d6b08-predictor-serving-cert\""
Jul 10 18:50:37.068614 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.068590    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"]
Jul 10 18:50:37.079493 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.079465    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"]
Jul 10 18:50:37.079746 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.079713    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container" containerID="cri-o://57c705f22496bc9fb9b92c3d3022c213a5461d288342cc2634338d83a7ef2adc" gracePeriod=30
Jul 10 18:50:37.079902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.079768    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kube-rbac-proxy" containerID="cri-o://cebe2a0cc44ae8ec54e7cae52a339d2f7b5e03d049a1b1ed61595854642f0d42" gracePeriod=30
Jul 10 18:50:37.136191 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.136148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c1737f-3369-4d97-a7b4-e021e0d034bd-proxy-tls\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.136376 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.136239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8nnc\" (UniqueName: \"kubernetes.io/projected/a3c1737f-3369-4d97-a7b4-e021e0d034bd-kube-api-access-c8nnc\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.136376 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.136267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c1737f-3369-4d97-a7b4-e021e0d034bd-success-200-isvc-d6b08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.144423 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.144395    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"]
Jul 10 18:50:37.147812 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.147797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.150967 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.150927    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d6b08-predictor-serving-cert\""
Jul 10 18:50:37.150967 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.150943    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d6b08-kube-rbac-proxy-sar-config\""
Jul 10 18:50:37.158732 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.158711    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"]
Jul 10 18:50:37.210780 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.210748    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"]
Jul 10 18:50:37.211017 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.210990    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container" containerID="cri-o://c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480" gracePeriod=30
Jul 10 18:50:37.211136 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.211033    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kube-rbac-proxy" containerID="cri-o://a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1" gracePeriod=30
Jul 10 18:50:37.237110 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.237070    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e720215f-d2fa-456b-9df8-40781dba0c7a-error-404-isvc-d6b08-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.237233 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.237151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c1737f-3369-4d97-a7b4-e021e0d034bd-proxy-tls\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.237233 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.237203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e720215f-d2fa-456b-9df8-40781dba0c7a-proxy-tls\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.237302 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.237228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5rp\" (UniqueName: \"kubernetes.io/projected/e720215f-d2fa-456b-9df8-40781dba0c7a-kube-api-access-xn5rp\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.237302 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.237271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8nnc\" (UniqueName: \"kubernetes.io/projected/a3c1737f-3369-4d97-a7b4-e021e0d034bd-kube-api-access-c8nnc\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.237302 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.237291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c1737f-3369-4d97-a7b4-e021e0d034bd-success-200-isvc-d6b08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.237901 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.237876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c1737f-3369-4d97-a7b4-e021e0d034bd-success-200-isvc-d6b08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.240005 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.239979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c1737f-3369-4d97-a7b4-e021e0d034bd-proxy-tls\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.249913 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.249886    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8nnc\" (UniqueName: \"kubernetes.io/projected/a3c1737f-3369-4d97-a7b4-e021e0d034bd-kube-api-access-c8nnc\") pod \"success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") " pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.338239 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.338200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e720215f-d2fa-456b-9df8-40781dba0c7a-error-404-isvc-d6b08-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.338468 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.338328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e720215f-d2fa-456b-9df8-40781dba0c7a-proxy-tls\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.338468 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.338359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5rp\" (UniqueName: \"kubernetes.io/projected/e720215f-d2fa-456b-9df8-40781dba0c7a-kube-api-access-xn5rp\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.339080 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.339055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e720215f-d2fa-456b-9df8-40781dba0c7a-error-404-isvc-d6b08-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.341239 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.341177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e720215f-d2fa-456b-9df8-40781dba0c7a-proxy-tls\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.347227 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.347177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5rp\" (UniqueName: \"kubernetes.io/projected/e720215f-d2fa-456b-9df8-40781dba0c7a-kube-api-access-xn5rp\") pod \"error-404-isvc-d6b08-predictor-6878886558-5kjcn\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") " pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.369922 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.369888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.458821 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.458762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.527938 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.527825    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"]
Jul 10 18:50:37.530411 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:50:37.530377    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c1737f_3369_4d97_a7b4_e021e0d034bd.slice/crio-78aed6428a6c7ca332df5cc6a169058bb1acb51c6478b6ab130c83d2eb96fa31 WatchSource:0}: Error finding container 78aed6428a6c7ca332df5cc6a169058bb1acb51c6478b6ab130c83d2eb96fa31: Status 404 returned error can't find the container with id 78aed6428a6c7ca332df5cc6a169058bb1acb51c6478b6ab130c83d2eb96fa31
Jul 10 18:50:37.605324 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.605267    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"]
Jul 10 18:50:37.607475 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:50:37.607449    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode720215f_d2fa_456b_9df8_40781dba0c7a.slice/crio-e88b2f4b67c7a0cadffc9833e7873f3a2789095f0d5d1741de14301ed46db37f WatchSource:0}: Error finding container e88b2f4b67c7a0cadffc9833e7873f3a2789095f0d5d1741de14301ed46db37f: Status 404 returned error can't find the container with id e88b2f4b67c7a0cadffc9833e7873f3a2789095f0d5d1741de14301ed46db37f
Jul 10 18:50:37.622702 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.622651    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 10 18:50:37.938782 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.938724    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:37.978147 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.978111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" event={"ID":"e720215f-d2fa-456b-9df8-40781dba0c7a","Type":"ContainerStarted","Data":"e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33"}
Jul 10 18:50:37.978308 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.978167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" event={"ID":"e720215f-d2fa-456b-9df8-40781dba0c7a","Type":"ContainerStarted","Data":"671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f"}
Jul 10 18:50:37.978308 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.978204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.978308 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.978239    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:37.978308 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.978267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" event={"ID":"e720215f-d2fa-456b-9df8-40781dba0c7a","Type":"ContainerStarted","Data":"e88b2f4b67c7a0cadffc9833e7873f3a2789095f0d5d1741de14301ed46db37f"}
Jul 10 18:50:37.979650 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.979616    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 18:50:37.980006 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.979983    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" event={"ID":"a3c1737f-3369-4d97-a7b4-e021e0d034bd","Type":"ContainerStarted","Data":"15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1"}
Jul 10 18:50:37.980116 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.980036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" event={"ID":"a3c1737f-3369-4d97-a7b4-e021e0d034bd","Type":"ContainerStarted","Data":"f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314"}
Jul 10 18:50:37.980116 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.980063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" event={"ID":"a3c1737f-3369-4d97-a7b4-e021e0d034bd","Type":"ContainerStarted","Data":"78aed6428a6c7ca332df5cc6a169058bb1acb51c6478b6ab130c83d2eb96fa31"}
Jul 10 18:50:37.980116 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.980094    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:37.981719 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.981695    2565 generic.go:356] "Generic (PLEG): container finished" podID="a66ce281-c2d2-41c1-803c-303353635f5d" containerID="a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1" exitCode=2
Jul 10 18:50:37.981875 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.981765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" event={"ID":"a66ce281-c2d2-41c1-803c-303353635f5d","Type":"ContainerDied","Data":"a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1"}
Jul 10 18:50:37.983455 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.983435    2565 generic.go:356] "Generic (PLEG): container finished" podID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerID="cebe2a0cc44ae8ec54e7cae52a339d2f7b5e03d049a1b1ed61595854642f0d42" exitCode=2
Jul 10 18:50:37.983536 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:37.983464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" event={"ID":"62e9f59e-c117-4fb6-bc58-e0569b8f9ade","Type":"ContainerDied","Data":"cebe2a0cc44ae8ec54e7cae52a339d2f7b5e03d049a1b1ed61595854642f0d42"}
Jul 10 18:50:38.005558 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:38.005499    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podStartSLOduration=1.005456685 podStartE2EDuration="1.005456685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:50:38.003250099 +0000 UTC m=+619.301592169" watchObservedRunningTime="2026-07-10 18:50:38.005456685 +0000 UTC m=+619.303798755"
Jul 10 18:50:38.033923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:38.033874    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podStartSLOduration=1.033858336 podStartE2EDuration="1.033858336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:50:38.032180801 +0000 UTC m=+619.330522889" watchObservedRunningTime="2026-07-10 18:50:38.033858336 +0000 UTC m=+619.332200405"
Jul 10 18:50:38.250076 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:38.249974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:38.987898 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:38.987844    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 18:50:38.988367 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:38.987968    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:38.989227 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:38.989201    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 18:50:39.991605 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:39.991525    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 18:50:40.565160 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.565129    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:50:40.664380 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.664340    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66ce281-c2d2-41c1-803c-303353635f5d-error-404-isvc-ba55b-kube-rbac-proxy-sar-config\") pod \"a66ce281-c2d2-41c1-803c-303353635f5d\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") "
Jul 10 18:50:40.664633 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.664416    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66ce281-c2d2-41c1-803c-303353635f5d-proxy-tls\") pod \"a66ce281-c2d2-41c1-803c-303353635f5d\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") "
Jul 10 18:50:40.664633 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.664513    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/a66ce281-c2d2-41c1-803c-303353635f5d-kube-api-access-wckt9\") pod \"a66ce281-c2d2-41c1-803c-303353635f5d\" (UID: \"a66ce281-c2d2-41c1-803c-303353635f5d\") "
Jul 10 18:50:40.664777 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.664748    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66ce281-c2d2-41c1-803c-303353635f5d-error-404-isvc-ba55b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-ba55b-kube-rbac-proxy-sar-config") pod "a66ce281-c2d2-41c1-803c-303353635f5d" (UID: "a66ce281-c2d2-41c1-803c-303353635f5d"). InnerVolumeSpecName "error-404-isvc-ba55b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:50:40.667349 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.667319    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66ce281-c2d2-41c1-803c-303353635f5d-kube-api-access-wckt9" (OuterVolumeSpecName: "kube-api-access-wckt9") pod "a66ce281-c2d2-41c1-803c-303353635f5d" (UID: "a66ce281-c2d2-41c1-803c-303353635f5d"). InnerVolumeSpecName "kube-api-access-wckt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:50:40.667471 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.667358    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66ce281-c2d2-41c1-803c-303353635f5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a66ce281-c2d2-41c1-803c-303353635f5d" (UID: "a66ce281-c2d2-41c1-803c-303353635f5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:50:40.765110 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.765075    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/a66ce281-c2d2-41c1-803c-303353635f5d-kube-api-access-wckt9\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:50:40.765110 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.765109    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66ce281-c2d2-41c1-803c-303353635f5d-error-404-isvc-ba55b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:50:40.765296 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.765125    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66ce281-c2d2-41c1-803c-303353635f5d-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:50:40.997525 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.997489    2565 generic.go:356] "Generic (PLEG): container finished" podID="a66ce281-c2d2-41c1-803c-303353635f5d" containerID="c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480" exitCode=0
Jul 10 18:50:40.998004 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.997561    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"
Jul 10 18:50:40.998004 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.997576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" event={"ID":"a66ce281-c2d2-41c1-803c-303353635f5d","Type":"ContainerDied","Data":"c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480"}
Jul 10 18:50:40.998004 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.997695    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng" event={"ID":"a66ce281-c2d2-41c1-803c-303353635f5d","Type":"ContainerDied","Data":"6cc55df1b3df63d6107b988555dba303a6751587f6fed88db9292a572356d230"}
Jul 10 18:50:40.998004 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.997740    2565 scope.go:117] "RemoveContainer" containerID="a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1"
Jul 10 18:50:40.999792 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.999767    2565 generic.go:356] "Generic (PLEG): container finished" podID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerID="57c705f22496bc9fb9b92c3d3022c213a5461d288342cc2634338d83a7ef2adc" exitCode=0
Jul 10 18:50:40.999926 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:40.999831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" event={"ID":"62e9f59e-c117-4fb6-bc58-e0569b8f9ade","Type":"ContainerDied","Data":"57c705f22496bc9fb9b92c3d3022c213a5461d288342cc2634338d83a7ef2adc"}
Jul 10 18:50:41.007513 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.007496    2565 scope.go:117] "RemoveContainer" containerID="c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480"
Jul 10 18:50:41.016109 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.016093    2565 scope.go:117] "RemoveContainer" containerID="a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1"
Jul 10 18:50:41.016342 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:50:41.016326    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1\": container with ID starting with a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1 not found: ID does not exist" containerID="a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1"
Jul 10 18:50:41.016426 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.016355    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1"} err="failed to get container status \"a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1\": rpc error: code = NotFound desc = could not find container \"a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1\": container with ID starting with a1fbe8ff5505484ac52fa48637308e439f331c8d2a223d5e10a6e447605a29d1 not found: ID does not exist"
Jul 10 18:50:41.016426 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.016379    2565 scope.go:117] "RemoveContainer" containerID="c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480"
Jul 10 18:50:41.016630 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:50:41.016615    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480\": container with ID starting with c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480 not found: ID does not exist" containerID="c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480"
Jul 10 18:50:41.016679 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.016637    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480"} err="failed to get container status \"c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480\": rpc error: code = NotFound desc = could not find container \"c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480\": container with ID starting with c40ed928d6eb124c506f10eeace6a90ebf27a6c1583c0dcc376892cae6bbe480 not found: ID does not exist"
Jul 10 18:50:41.016852 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.016836    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:50:41.026267 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.026245    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"]
Jul 10 18:50:41.029233 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.029213    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ba55b-predictor-578475f44-lgsng"]
Jul 10 18:50:41.067563 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.067539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls\") pod \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") "
Jul 10 18:50:41.067689 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.067570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-success-200-isvc-ba55b-kube-rbac-proxy-sar-config\") pod \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") "
Jul 10 18:50:41.067689 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.067599    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdhs6\" (UniqueName: \"kubernetes.io/projected/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-kube-api-access-kdhs6\") pod \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\" (UID: \"62e9f59e-c117-4fb6-bc58-e0569b8f9ade\") "
Jul 10 18:50:41.068006 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.067978    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-success-200-isvc-ba55b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-ba55b-kube-rbac-proxy-sar-config") pod "62e9f59e-c117-4fb6-bc58-e0569b8f9ade" (UID: "62e9f59e-c117-4fb6-bc58-e0569b8f9ade"). InnerVolumeSpecName "success-200-isvc-ba55b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:50:41.069904 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.069880    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-kube-api-access-kdhs6" (OuterVolumeSpecName: "kube-api-access-kdhs6") pod "62e9f59e-c117-4fb6-bc58-e0569b8f9ade" (UID: "62e9f59e-c117-4fb6-bc58-e0569b8f9ade"). InnerVolumeSpecName "kube-api-access-kdhs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:50:41.070139 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.070119    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62e9f59e-c117-4fb6-bc58-e0569b8f9ade" (UID: "62e9f59e-c117-4fb6-bc58-e0569b8f9ade"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:50:41.169132 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.169103    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:50:41.169300 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.169151    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-ba55b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-success-200-isvc-ba55b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:50:41.169300 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.169164    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdhs6\" (UniqueName: \"kubernetes.io/projected/62e9f59e-c117-4fb6-bc58-e0569b8f9ade-kube-api-access-kdhs6\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:50:41.273306 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:41.273229    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" path="/var/lib/kubelet/pods/a66ce281-c2d2-41c1-803c-303353635f5d/volumes"
Jul 10 18:50:42.006477 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:42.006444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh" event={"ID":"62e9f59e-c117-4fb6-bc58-e0569b8f9ade","Type":"ContainerDied","Data":"4630f02a1c324058a886d2f5a0dd5759a4555b9b90ef068044476cb544800639"}
Jul 10 18:50:42.006889 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:42.006485    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"
Jul 10 18:50:42.006889 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:42.006509    2565 scope.go:117] "RemoveContainer" containerID="cebe2a0cc44ae8ec54e7cae52a339d2f7b5e03d049a1b1ed61595854642f0d42"
Jul 10 18:50:42.014714 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:42.014695    2565 scope.go:117] "RemoveContainer" containerID="57c705f22496bc9fb9b92c3d3022c213a5461d288342cc2634338d83a7ef2adc"
Jul 10 18:50:42.024460 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:42.024438    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"]
Jul 10 18:50:42.028067 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:42.028042    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ba55b-predictor-575d4874b7-tx7wh"]
Jul 10 18:50:42.938280 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:42.938243    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:43.273163 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:43.273085    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" path="/var/lib/kubelet/pods/62e9f59e-c117-4fb6-bc58-e0569b8f9ade/volumes"
Jul 10 18:50:43.992868 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:43.992840    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:50:43.993425 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:43.993388    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 18:50:44.996693 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:44.996661    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:50:44.997304 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:44.997277    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 18:50:47.938778 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:47.938736    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:47.939178 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:47.938849    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:50:51.786111 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:51.786066    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:51.838782 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:51.838740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:50:52.938816 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:52.938777    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:53.994436 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:53.994399    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 18:50:54.997242 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:54.997194    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 18:50:57.938718 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:50:57.938670    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:02.726848 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.726803    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"]
Jul 10 18:51:02.727380 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727358    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kube-rbac-proxy"
Jul 10 18:51:02.727455 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727385    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kube-rbac-proxy"
Jul 10 18:51:02.727455 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727398    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container"
Jul 10 18:51:02.727455 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727406    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container"
Jul 10 18:51:02.727455 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727430    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container"
Jul 10 18:51:02.727651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727472    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container"
Jul 10 18:51:02.727651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727491    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kube-rbac-proxy"
Jul 10 18:51:02.727651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727501    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kube-rbac-proxy"
Jul 10 18:51:02.727651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727595    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kserve-container"
Jul 10 18:51:02.727651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727609    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kserve-container"
Jul 10 18:51:02.727651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727619    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66ce281-c2d2-41c1-803c-303353635f5d" containerName="kube-rbac-proxy"
Jul 10 18:51:02.727651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.727630    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e9f59e-c117-4fb6-bc58-e0569b8f9ade" containerName="kube-rbac-proxy"
Jul 10 18:51:02.734031 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.734007    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:02.737006 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.736695    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 10 18:51:02.737006 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.736938    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 10 18:51:02.738135 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.738103    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:02.738866 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.738838    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"]
Jul 10 18:51:02.856105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.856058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acbc024c-cfc7-4829-940c-29b52f0d20e5-proxy-tls\") pod \"model-chainer-6b7b849495-2wvqc\" (UID: \"acbc024c-cfc7-4829-940c-29b52f0d20e5\") " pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:02.856105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.856109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acbc024c-cfc7-4829-940c-29b52f0d20e5-openshift-service-ca-bundle\") pod \"model-chainer-6b7b849495-2wvqc\" (UID: \"acbc024c-cfc7-4829-940c-29b52f0d20e5\") " pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:02.900248 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.900190    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:02.938589 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.938550    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:02.957529 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.957495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acbc024c-cfc7-4829-940c-29b52f0d20e5-proxy-tls\") pod \"model-chainer-6b7b849495-2wvqc\" (UID: \"acbc024c-cfc7-4829-940c-29b52f0d20e5\") " pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:02.957728 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.957545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acbc024c-cfc7-4829-940c-29b52f0d20e5-openshift-service-ca-bundle\") pod \"model-chainer-6b7b849495-2wvqc\" (UID: \"acbc024c-cfc7-4829-940c-29b52f0d20e5\") " pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:02.958210 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.958190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acbc024c-cfc7-4829-940c-29b52f0d20e5-openshift-service-ca-bundle\") pod \"model-chainer-6b7b849495-2wvqc\" (UID: \"acbc024c-cfc7-4829-940c-29b52f0d20e5\") " pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:02.960178 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:02.960158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acbc024c-cfc7-4829-940c-29b52f0d20e5-proxy-tls\") pod \"model-chainer-6b7b849495-2wvqc\" (UID: \"acbc024c-cfc7-4829-940c-29b52f0d20e5\") " pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:03.046115 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:03.046014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:03.184281 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:03.184174    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"]
Jul 10 18:51:03.186915 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:51:03.186881    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbc024c_cfc7_4829_940c_29b52f0d20e5.slice/crio-32ddc48c5fad4f976308274b8c7a8360ffdf4430419c2dd94e203cf81b19a581 WatchSource:0}: Error finding container 32ddc48c5fad4f976308274b8c7a8360ffdf4430419c2dd94e203cf81b19a581: Status 404 returned error can't find the container with id 32ddc48c5fad4f976308274b8c7a8360ffdf4430419c2dd94e203cf81b19a581
Jul 10 18:51:03.993760 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:03.993714    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 18:51:04.102521 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:04.102484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" event={"ID":"acbc024c-cfc7-4829-940c-29b52f0d20e5","Type":"ContainerStarted","Data":"06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be"}
Jul 10 18:51:04.102690 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:04.102535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" event={"ID":"acbc024c-cfc7-4829-940c-29b52f0d20e5","Type":"ContainerStarted","Data":"32ddc48c5fad4f976308274b8c7a8360ffdf4430419c2dd94e203cf81b19a581"}
Jul 10 18:51:04.102690 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:04.102556    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:04.120899 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:04.120843    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" podStartSLOduration=2.120827978 podStartE2EDuration="2.120827978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:04.119868607 +0000 UTC m=+645.418210653" watchObservedRunningTime="2026-07-10 18:51:04.120827978 +0000 UTC m=+645.419170047"
Jul 10 18:51:04.997438 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:04.997399    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 18:51:06.980582 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:06.980554    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:51:07.094273 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.094180    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adcdc57d-f1b6-4e05-a1bc-ec031351202e-openshift-service-ca-bundle\") pod \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") "
Jul 10 18:51:07.094441 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.094287    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls\") pod \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\" (UID: \"adcdc57d-f1b6-4e05-a1bc-ec031351202e\") "
Jul 10 18:51:07.094549 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.094523    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcdc57d-f1b6-4e05-a1bc-ec031351202e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "adcdc57d-f1b6-4e05-a1bc-ec031351202e" (UID: "adcdc57d-f1b6-4e05-a1bc-ec031351202e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:51:07.096520 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.096499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "adcdc57d-f1b6-4e05-a1bc-ec031351202e" (UID: "adcdc57d-f1b6-4e05-a1bc-ec031351202e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:51:07.118109 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.118080    2565 generic.go:356] "Generic (PLEG): container finished" podID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerID="a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1" exitCode=0
Jul 10 18:51:07.118238 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.118126    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"
Jul 10 18:51:07.118238 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.118168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" event={"ID":"adcdc57d-f1b6-4e05-a1bc-ec031351202e","Type":"ContainerDied","Data":"a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1"}
Jul 10 18:51:07.118238 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.118219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5" event={"ID":"adcdc57d-f1b6-4e05-a1bc-ec031351202e","Type":"ContainerDied","Data":"4ad15db448321085140b7f70cb53df3e4f7e460bdf776ec9bc607795a479726f"}
Jul 10 18:51:07.118367 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.118241    2565 scope.go:117] "RemoveContainer" containerID="a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1"
Jul 10 18:51:07.126924 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.126904    2565 scope.go:117] "RemoveContainer" containerID="a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1"
Jul 10 18:51:07.127225 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:51:07.127207    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1\": container with ID starting with a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1 not found: ID does not exist" containerID="a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1"
Jul 10 18:51:07.127285 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.127236    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1"} err="failed to get container status \"a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1\": rpc error: code = NotFound desc = could not find container \"a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1\": container with ID starting with a17b2ef5cd2bb2260651160ecb79a976cf2ac167c419cd9edfcacde3804aeef1 not found: ID does not exist"
Jul 10 18:51:07.138354 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.138329    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"]
Jul 10 18:51:07.141869 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.141846    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ba55b-7558d56dd-crnz5"]
Jul 10 18:51:07.194799 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.194768    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adcdc57d-f1b6-4e05-a1bc-ec031351202e-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:07.194799 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.194795    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adcdc57d-f1b6-4e05-a1bc-ec031351202e-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:07.273256 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:07.273225    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" path="/var/lib/kubelet/pods/adcdc57d-f1b6-4e05-a1bc-ec031351202e/volumes"
Jul 10 18:51:10.113123 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:10.113092    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:12.812347 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:12.812312    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"]
Jul 10 18:51:12.812849 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:12.812522    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer" containerID="cri-o://06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be" gracePeriod=30
Jul 10 18:51:12.841704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:12.841670    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:12.940102 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:12.940061    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"]
Jul 10 18:51:12.940368 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:12.940341    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container" containerID="cri-o://502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151" gracePeriod=30
Jul 10 18:51:12.940452 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:12.940392    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kube-rbac-proxy" containerID="cri-o://2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0" gracePeriod=30
Jul 10 18:51:12.967093 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:12.967065    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:13.000181 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.000146    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"]
Jul 10 18:51:13.000450 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.000418    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" containerID="cri-o://4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab" gracePeriod=30
Jul 10 18:51:13.000588 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.000464    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kube-rbac-proxy" containerID="cri-o://016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae" gracePeriod=30
Jul 10 18:51:13.011643 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.011619    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"]
Jul 10 18:51:13.012067 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.012052    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b"
Jul 10 18:51:13.012106 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.012069    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b"
Jul 10 18:51:13.012164 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.012152    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="adcdc57d-f1b6-4e05-a1bc-ec031351202e" containerName="switch-graph-ba55b"
Jul 10 18:51:13.015361 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.015344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.017809 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.017790    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-34ebc-predictor-serving-cert\""
Jul 10 18:51:13.017912 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.017853    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-34ebc-kube-rbac-proxy-sar-config\""
Jul 10 18:51:13.028029 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.028004    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"]
Jul 10 18:51:13.064661 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.064583    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:13.105931 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.105897    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"]
Jul 10 18:51:13.109904 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.109879    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.112975 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.112806    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-34ebc-kube-rbac-proxy-sar-config\""
Jul 10 18:51:13.112975 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.112825    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-34ebc-predictor-serving-cert\""
Jul 10 18:51:13.120998 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.120882    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"]
Jul 10 18:51:13.142230 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.142206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1970204-9768-4dec-9cbc-3e47d4620f18-proxy-tls\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.142361 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.142256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgxwc\" (UniqueName: \"kubernetes.io/projected/a1970204-9768-4dec-9cbc-3e47d4620f18-kube-api-access-sgxwc\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.142361 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.142291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1970204-9768-4dec-9cbc-3e47d4620f18-success-200-isvc-34ebc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.149797 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.149764    2565 generic.go:356] "Generic (PLEG): container finished" podID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerID="2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0" exitCode=2
Jul 10 18:51:13.149940 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.149840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" event={"ID":"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2","Type":"ContainerDied","Data":"2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0"}
Jul 10 18:51:13.152428 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.152400    2565 generic.go:356] "Generic (PLEG): container finished" podID="1e44c877-3e7a-488f-9e44-5277532972c7" containerID="016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae" exitCode=2
Jul 10 18:51:13.152547 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.152433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" event={"ID":"1e44c877-3e7a-488f-9e44-5277532972c7","Type":"ContainerDied","Data":"016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae"}
Jul 10 18:51:13.154995 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.154975    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:13.213600 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.213571    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:13.243467 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.243436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-proxy-tls\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.243467 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.243472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526wh\" (UniqueName: \"kubernetes.io/projected/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-kube-api-access-526wh\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.243724 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.243501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1970204-9768-4dec-9cbc-3e47d4620f18-proxy-tls\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.243724 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.243523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-error-404-isvc-34ebc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.243724 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.243612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxwc\" (UniqueName: \"kubernetes.io/projected/a1970204-9768-4dec-9cbc-3e47d4620f18-kube-api-access-sgxwc\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.243724 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.243660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1970204-9768-4dec-9cbc-3e47d4620f18-success-200-isvc-34ebc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.244372 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.244309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1970204-9768-4dec-9cbc-3e47d4620f18-success-200-isvc-34ebc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.246057 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.246039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1970204-9768-4dec-9cbc-3e47d4620f18-proxy-tls\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.251717 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.251688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgxwc\" (UniqueName: \"kubernetes.io/projected/a1970204-9768-4dec-9cbc-3e47d4620f18-kube-api-access-sgxwc\") pod \"success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") " pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.266545 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.266521    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:13.325757 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.325684    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:13.344947 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.344910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-proxy-tls\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.345086 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.344996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526wh\" (UniqueName: \"kubernetes.io/projected/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-kube-api-access-526wh\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.345086 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.345047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-error-404-isvc-34ebc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.345715 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.345691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-error-404-isvc-34ebc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.347677 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.347642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-proxy-tls\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.355023 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.354983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-526wh\" (UniqueName: \"kubernetes.io/projected/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-kube-api-access-526wh\") pod \"error-404-isvc-34ebc-predictor-59d677957b-wtgd2\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") " pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.424312 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.424281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:13.461646 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.461619    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"]
Jul 10 18:51:13.463134 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:51:13.463098    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1970204_9768_4dec_9cbc_3e47d4620f18.slice/crio-3553b60b2fe4de1dacf1c5537f27feb5d4677694e4761b2c1da6eac973be39a8 WatchSource:0}: Error finding container 3553b60b2fe4de1dacf1c5537f27feb5d4677694e4761b2c1da6eac973be39a8: Status 404 returned error can't find the container with id 3553b60b2fe4de1dacf1c5537f27feb5d4677694e4761b2c1da6eac973be39a8
Jul 10 18:51:13.568767 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.568652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"]
Jul 10 18:51:13.573063 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:51:13.573024    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ac9ffb_3d16_443b_a7d0_9d43a1e82dc8.slice/crio-faa26ba7c1f25b575b64aa23d832d4174c10c0c08810ebecdd6881205e2f9722 WatchSource:0}: Error finding container faa26ba7c1f25b575b64aa23d832d4174c10c0c08810ebecdd6881205e2f9722: Status 404 returned error can't find the container with id faa26ba7c1f25b575b64aa23d832d4174c10c0c08810ebecdd6881205e2f9722
Jul 10 18:51:13.994355 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:13.994312    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 18:51:14.160528 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.160487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" event={"ID":"a1970204-9768-4dec-9cbc-3e47d4620f18","Type":"ContainerStarted","Data":"86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814"}
Jul 10 18:51:14.160697 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.160594    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:14.160697 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.160624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" event={"ID":"a1970204-9768-4dec-9cbc-3e47d4620f18","Type":"ContainerStarted","Data":"3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1"}
Jul 10 18:51:14.160697 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.160651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" event={"ID":"a1970204-9768-4dec-9cbc-3e47d4620f18","Type":"ContainerStarted","Data":"3553b60b2fe4de1dacf1c5537f27feb5d4677694e4761b2c1da6eac973be39a8"}
Jul 10 18:51:14.160697 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.160679    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:14.162393 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.162353    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 18:51:14.162519 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.162469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" event={"ID":"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8","Type":"ContainerStarted","Data":"778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090"}
Jul 10 18:51:14.162592 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.162536    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:14.162592 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.162570    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:14.162684 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.162600    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" event={"ID":"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8","Type":"ContainerStarted","Data":"7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36"}
Jul 10 18:51:14.162684 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.162629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" event={"ID":"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8","Type":"ContainerStarted","Data":"faa26ba7c1f25b575b64aa23d832d4174c10c0c08810ebecdd6881205e2f9722"}
Jul 10 18:51:14.163704 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.163679    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 18:51:14.179994 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.179928    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podStartSLOduration=2.179913695 podStartE2EDuration="2.179913695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:14.178364558 +0000 UTC m=+655.476706625" watchObservedRunningTime="2026-07-10 18:51:14.179913695 +0000 UTC m=+655.478255763"
Jul 10 18:51:14.195620 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.195575    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podStartSLOduration=1.1955642690000001 podStartE2EDuration="1.195564269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:14.195078968 +0000 UTC m=+655.493421037" watchObservedRunningTime="2026-07-10 18:51:14.195564269 +0000 UTC m=+655.493906337"
Jul 10 18:51:14.998113 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:14.998071    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 18:51:15.111663 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:15.111614    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:15.168106 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:15.168061    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 18:51:15.168289 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:15.168181    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 18:51:15.268418 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:15.268330    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 18:51:15.664226 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:15.664185    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 10 18:51:15.664399 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:15.664186    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 10 18:51:18.703184 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.703125    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:51:18.746348 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.746324    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:51:18.797533 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.797499    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-proxy-tls\") pod \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") "
Jul 10 18:51:18.797707 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.797551    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kserve-provision-location\") pod \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") "
Jul 10 18:51:18.797773 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.797730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jnvv\" (UniqueName: \"kubernetes.io/projected/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kube-api-access-8jnvv\") pod \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") "
Jul 10 18:51:18.797773 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.797768    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\" (UID: \"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2\") "
Jul 10 18:51:18.797912 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.797878    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" (UID: "cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:51:18.798154 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.798133    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kserve-provision-location\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:18.798154 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.798125    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config") pod "cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" (UID: "cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2"). InnerVolumeSpecName "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:51:18.799727 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.799706    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" (UID: "cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:51:18.799800 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.799780    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kube-api-access-8jnvv" (OuterVolumeSpecName: "kube-api-access-8jnvv") pod "cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" (UID: "cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2"). InnerVolumeSpecName "kube-api-access-8jnvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:51:18.899381 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899338    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e44c877-3e7a-488f-9e44-5277532972c7-kserve-provision-location\") pod \"1e44c877-3e7a-488f-9e44-5277532972c7\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") "
Jul 10 18:51:18.899549 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxq5f\" (UniqueName: \"kubernetes.io/projected/1e44c877-3e7a-488f-9e44-5277532972c7-kube-api-access-pxq5f\") pod \"1e44c877-3e7a-488f-9e44-5277532972c7\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") "
Jul 10 18:51:18.899549 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899495    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e44c877-3e7a-488f-9e44-5277532972c7-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"1e44c877-3e7a-488f-9e44-5277532972c7\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") "
Jul 10 18:51:18.899549 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899516    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls\") pod \"1e44c877-3e7a-488f-9e44-5277532972c7\" (UID: \"1e44c877-3e7a-488f-9e44-5277532972c7\") "
Jul 10 18:51:18.899668 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899633    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:18.899668 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899643    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jnvv\" (UniqueName: \"kubernetes.io/projected/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-kube-api-access-8jnvv\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:18.899668 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899655    2565 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:18.899780 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e44c877-3e7a-488f-9e44-5277532972c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e44c877-3e7a-488f-9e44-5277532972c7" (UID: "1e44c877-3e7a-488f-9e44-5277532972c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:51:18.899882 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.899848    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e44c877-3e7a-488f-9e44-5277532972c7-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config") pod "1e44c877-3e7a-488f-9e44-5277532972c7" (UID: "1e44c877-3e7a-488f-9e44-5277532972c7"). InnerVolumeSpecName "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:51:18.901827 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.901802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e44c877-3e7a-488f-9e44-5277532972c7-kube-api-access-pxq5f" (OuterVolumeSpecName: "kube-api-access-pxq5f") pod "1e44c877-3e7a-488f-9e44-5277532972c7" (UID: "1e44c877-3e7a-488f-9e44-5277532972c7"). InnerVolumeSpecName "kube-api-access-pxq5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:51:18.901890 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:18.901830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e44c877-3e7a-488f-9e44-5277532972c7" (UID: "1e44c877-3e7a-488f-9e44-5277532972c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:51:19.000220 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.000139    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxq5f\" (UniqueName: \"kubernetes.io/projected/1e44c877-3e7a-488f-9e44-5277532972c7-kube-api-access-pxq5f\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:19.000220 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.000166    2565 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e44c877-3e7a-488f-9e44-5277532972c7-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:19.000220 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.000176    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e44c877-3e7a-488f-9e44-5277532972c7-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:19.000220 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.000186    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e44c877-3e7a-488f-9e44-5277532972c7-kserve-provision-location\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:19.189377 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.189341    2565 generic.go:356] "Generic (PLEG): container finished" podID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerID="502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151" exitCode=0
Jul 10 18:51:19.189531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.189408    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"
Jul 10 18:51:19.189531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.189425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" event={"ID":"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2","Type":"ContainerDied","Data":"502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151"}
Jul 10 18:51:19.189531 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.189505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6" event={"ID":"cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2","Type":"ContainerDied","Data":"f4ab4f2cfa399d2de28724363190e9f239a1112322d8d1e183c949d5853e1b98"}
Jul 10 18:51:19.189667 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.189547    2565 scope.go:117] "RemoveContainer" containerID="2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0"
Jul 10 18:51:19.191564 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.191541    2565 generic.go:356] "Generic (PLEG): container finished" podID="1e44c877-3e7a-488f-9e44-5277532972c7" containerID="4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab" exitCode=0
Jul 10 18:51:19.191651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.191605    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"
Jul 10 18:51:19.191651 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.191622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" event={"ID":"1e44c877-3e7a-488f-9e44-5277532972c7","Type":"ContainerDied","Data":"4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab"}
Jul 10 18:51:19.191731 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.191683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx" event={"ID":"1e44c877-3e7a-488f-9e44-5277532972c7","Type":"ContainerDied","Data":"89edcd6349e77fd9897ea447b6a92f17b1c8f8cb9b4e86efc098f65c06490d8b"}
Jul 10 18:51:19.199398 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.199382    2565 scope.go:117] "RemoveContainer" containerID="502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151"
Jul 10 18:51:19.207460 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.207442    2565 scope.go:117] "RemoveContainer" containerID="c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198"
Jul 10 18:51:19.214501 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.214397    2565 scope.go:117] "RemoveContainer" containerID="2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0"
Jul 10 18:51:19.214930 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:51:19.214805    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0\": container with ID starting with 2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0 not found: ID does not exist" containerID="2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0"
Jul 10 18:51:19.214930 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.214840    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0"} err="failed to get container status \"2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0\": rpc error: code = NotFound desc = could not find container \"2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0\": container with ID starting with 2fbbdd7abce188c6f79dbe2c189b208204c1be8c4c249844e6c00e5c956f74b0 not found: ID does not exist"
Jul 10 18:51:19.214930 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.214883    2565 scope.go:117] "RemoveContainer" containerID="502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151"
Jul 10 18:51:19.215242 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:51:19.215216    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151\": container with ID starting with 502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151 not found: ID does not exist" containerID="502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151"
Jul 10 18:51:19.215321 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.215249    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151"} err="failed to get container status \"502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151\": rpc error: code = NotFound desc = could not find container \"502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151\": container with ID starting with 502a2716aac819ccf64b6b5bb7b61997d1ef372ce2ba78c095faa82028546151 not found: ID does not exist"
Jul 10 18:51:19.215321 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.215272    2565 scope.go:117] "RemoveContainer" containerID="c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198"
Jul 10 18:51:19.215627 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:51:19.215598    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198\": container with ID starting with c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198 not found: ID does not exist" containerID="c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198"
Jul 10 18:51:19.215750 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.215727    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198"} err="failed to get container status \"c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198\": rpc error: code = NotFound desc = could not find container \"c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198\": container with ID starting with c95aa30e068536a25184fac937d7a77736f6c209eb7007455191186d6542f198 not found: ID does not exist"
Jul 10 18:51:19.215844 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.215833    2565 scope.go:117] "RemoveContainer" containerID="016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae"
Jul 10 18:51:19.217246 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.217231    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"]
Jul 10 18:51:19.223086 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.223060    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-745b467f-g9mb6"]
Jul 10 18:51:19.224940 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.224925    2565 scope.go:117] "RemoveContainer" containerID="4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab"
Jul 10 18:51:19.232447 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.232426    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"]
Jul 10 18:51:19.232652 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.232637    2565 scope.go:117] "RemoveContainer" containerID="86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09"
Jul 10 18:51:19.236912 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.236891    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-54dd4fbd97-nzxvx"]
Jul 10 18:51:19.240353 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.240338    2565 scope.go:117] "RemoveContainer" containerID="016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae"
Jul 10 18:51:19.240629 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:51:19.240613    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae\": container with ID starting with 016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae not found: ID does not exist" containerID="016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae"
Jul 10 18:51:19.240678 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.240634    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae"} err="failed to get container status \"016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae\": rpc error: code = NotFound desc = could not find container \"016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae\": container with ID starting with 016878788d10202d0af57211c86c4588fcf4522a23e545fab2d879e05bd2b8ae not found: ID does not exist"
Jul 10 18:51:19.240678 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.240650    2565 scope.go:117] "RemoveContainer" containerID="4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab"
Jul 10 18:51:19.240923 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:51:19.240902    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab\": container with ID starting with 4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab not found: ID does not exist" containerID="4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab"
Jul 10 18:51:19.241029 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.240926    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab"} err="failed to get container status \"4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab\": rpc error: code = NotFound desc = could not find container \"4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab\": container with ID starting with 4960b67437c6da220abb73be3382aa7eb7f281f225c17a30f8f0d1441cc20cab not found: ID does not exist"
Jul 10 18:51:19.241029 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.240943    2565 scope.go:117] "RemoveContainer" containerID="86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09"
Jul 10 18:51:19.241300 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:51:19.241286    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09\": container with ID starting with 86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09 not found: ID does not exist" containerID="86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09"
Jul 10 18:51:19.241389 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.241305    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09"} err="failed to get container status \"86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09\": rpc error: code = NotFound desc = could not find container \"86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09\": container with ID starting with 86012e38087340b3a834f3980c4750f503d019ebf76c84ed2a98ef41b5d1de09 not found: ID does not exist"
Jul 10 18:51:19.273261 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.273188    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" path="/var/lib/kubelet/pods/1e44c877-3e7a-488f-9e44-5277532972c7/volumes"
Jul 10 18:51:19.273670 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:19.273657    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" path="/var/lib/kubelet/pods/cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2/volumes"
Jul 10 18:51:20.111390 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:20.111353    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:20.174093 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:20.174066    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:51:20.174284 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:20.174167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:51:20.174601 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:20.174575    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 18:51:20.174700 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:20.174601    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 18:51:23.994214 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:23.994167    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 18:51:24.997775 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:24.997738    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 18:51:25.112794 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:25.112754    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:25.112974 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:25.112876    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:30.111487 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:30.111446    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:30.174920 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:30.174886    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 18:51:30.175100 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:30.174885    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 18:51:33.994148 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:33.994113    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 18:51:34.997839 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:34.997812    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 18:51:35.111943 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:35.111904    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:40.111892 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:40.111837    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:40.174995 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:40.174937    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 18:51:40.175186 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:40.174937    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 18:51:42.964764 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:42.964738    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:43.006835 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.006801    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acbc024c-cfc7-4829-940c-29b52f0d20e5-proxy-tls\") pod \"acbc024c-cfc7-4829-940c-29b52f0d20e5\" (UID: \"acbc024c-cfc7-4829-940c-29b52f0d20e5\") "
Jul 10 18:51:43.007038 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.006855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acbc024c-cfc7-4829-940c-29b52f0d20e5-openshift-service-ca-bundle\") pod \"acbc024c-cfc7-4829-940c-29b52f0d20e5\" (UID: \"acbc024c-cfc7-4829-940c-29b52f0d20e5\") "
Jul 10 18:51:43.007251 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.007224    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbc024c-cfc7-4829-940c-29b52f0d20e5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "acbc024c-cfc7-4829-940c-29b52f0d20e5" (UID: "acbc024c-cfc7-4829-940c-29b52f0d20e5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:51:43.009040 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.009008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbc024c-cfc7-4829-940c-29b52f0d20e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "acbc024c-cfc7-4829-940c-29b52f0d20e5" (UID: "acbc024c-cfc7-4829-940c-29b52f0d20e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:51:43.108152 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.108068    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acbc024c-cfc7-4829-940c-29b52f0d20e5-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:43.108152 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.108098    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acbc024c-cfc7-4829-940c-29b52f0d20e5-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 18:51:43.300397 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.300360    2565 generic.go:356] "Generic (PLEG): container finished" podID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerID="06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be" exitCode=0
Jul 10 18:51:43.300533 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.300406    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"
Jul 10 18:51:43.300533 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.300442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" event={"ID":"acbc024c-cfc7-4829-940c-29b52f0d20e5","Type":"ContainerDied","Data":"06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be"}
Jul 10 18:51:43.300533 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.300487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc" event={"ID":"acbc024c-cfc7-4829-940c-29b52f0d20e5","Type":"ContainerDied","Data":"32ddc48c5fad4f976308274b8c7a8360ffdf4430419c2dd94e203cf81b19a581"}
Jul 10 18:51:43.300533 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.300507    2565 scope.go:117] "RemoveContainer" containerID="06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be"
Jul 10 18:51:43.308947 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.308928    2565 scope.go:117] "RemoveContainer" containerID="06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be"
Jul 10 18:51:43.309218 ip-10-0-129-73 kubenswrapper[2565]: E0710 18:51:43.309199    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be\": container with ID starting with 06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be not found: ID does not exist" containerID="06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be"
Jul 10 18:51:43.309269 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.309228    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be"} err="failed to get container status \"06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be\": rpc error: code = NotFound desc = could not find container \"06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be\": container with ID starting with 06c1d7fe2f6775303971d65078a2261751aaef7f405b9eb5addb4b13405805be not found: ID does not exist"
Jul 10 18:51:43.316993 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.316966    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"]
Jul 10 18:51:43.318992 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:43.318969    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b7b849495-2wvqc"]
Jul 10 18:51:45.273548 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:45.273515    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" path="/var/lib/kubelet/pods/acbc024c-cfc7-4829-940c-29b52f0d20e5/volumes"
Jul 10 18:51:47.098475 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.098434    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"]
Jul 10 18:51:47.098975 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.098930    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container"
Jul 10 18:51:47.098975 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.098970    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.098993    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="storage-initializer"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099002    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="storage-initializer"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099023    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099033    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099044    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kube-rbac-proxy"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099054    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kube-rbac-proxy"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099066    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kube-rbac-proxy"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099076    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kube-rbac-proxy"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099100    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container"
Jul 10 18:51:47.099105 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099108    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container"
Jul 10 18:51:47.099552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099120    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="storage-initializer"
Jul 10 18:51:47.099552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099128    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="storage-initializer"
Jul 10 18:51:47.099552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099208    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kube-rbac-proxy"
Jul 10 18:51:47.099552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099222    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kserve-container"
Jul 10 18:51:47.099552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099232    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd7ff70-45d4-4a6e-a2c7-094d6ef757c2" containerName="kube-rbac-proxy"
Jul 10 18:51:47.099552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099245    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="acbc024c-cfc7-4829-940c-29b52f0d20e5" containerName="model-chainer"
Jul 10 18:51:47.099552 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.099253    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e44c877-3e7a-488f-9e44-5277532972c7" containerName="kserve-container"
Jul 10 18:51:47.103791 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.103769    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:47.105864 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.105841    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:47.106467 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.106451    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d6b08-serving-cert\""
Jul 10 18:51:47.106467 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.106461    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d6b08-kube-rbac-proxy-sar-config\""
Jul 10 18:51:47.116325 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.116301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"]
Jul 10 18:51:47.139499 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.139466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-openshift-service-ca-bundle\") pod \"switch-graph-d6b08-56b6fff7d5-vhpwk\" (UID: \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\") " pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:47.139679 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.139504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-proxy-tls\") pod \"switch-graph-d6b08-56b6fff7d5-vhpwk\" (UID: \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\") " pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:47.238470 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.238432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:51:47.240553 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.240529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-openshift-service-ca-bundle\") pod \"switch-graph-d6b08-56b6fff7d5-vhpwk\" (UID: \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\") " pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:47.240637 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.240563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-proxy-tls\") pod \"switch-graph-d6b08-56b6fff7d5-vhpwk\" (UID: \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\") " pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:47.241182 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.241159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-openshift-service-ca-bundle\") pod \"switch-graph-d6b08-56b6fff7d5-vhpwk\" (UID: \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\") " pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:47.243040 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.243024    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-proxy-tls\") pod \"switch-graph-d6b08-56b6fff7d5-vhpwk\" (UID: \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\") " pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:47.414480 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.414448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:47.546911 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:47.546876    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"]
Jul 10 18:51:47.549888 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:51:47.549854    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a3d8b0_b196_4247_8d71_c6118f8a6a4f.slice/crio-f1c02d7e76c401b30c3b2d27c4aa9287d22d63129f5316193ccf0c9cbe8a3558 WatchSource:0}: Error finding container f1c02d7e76c401b30c3b2d27c4aa9287d22d63129f5316193ccf0c9cbe8a3558: Status 404 returned error can't find the container with id f1c02d7e76c401b30c3b2d27c4aa9287d22d63129f5316193ccf0c9cbe8a3558
Jul 10 18:51:48.322128 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:48.322089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" event={"ID":"87a3d8b0-b196-4247-8d71-c6118f8a6a4f","Type":"ContainerStarted","Data":"65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b"}
Jul 10 18:51:48.322604 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:48.322156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" event={"ID":"87a3d8b0-b196-4247-8d71-c6118f8a6a4f","Type":"ContainerStarted","Data":"f1c02d7e76c401b30c3b2d27c4aa9287d22d63129f5316193ccf0c9cbe8a3558"}
Jul 10 18:51:48.322604 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:48.322193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:51:48.339003 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:48.338937    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" podStartSLOduration=1.338922162 podStartE2EDuration="1.338922162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:48.336698398 +0000 UTC m=+689.635040480" watchObservedRunningTime="2026-07-10 18:51:48.338922162 +0000 UTC m=+689.637264231"
Jul 10 18:51:50.174923 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:50.174887    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 18:51:50.175314 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:50.174887    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 18:51:54.333139 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:51:54.333106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 18:52:00.175110 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:00.175074    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 18:52:00.175673 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:00.175173    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 18:52:12.994967 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:12.994913    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"]
Jul 10 18:52:12.997632 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:12.997609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:13.000147 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.000125    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-34ebc-serving-cert\""
Jul 10 18:52:13.000250 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.000134    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-34ebc-kube-rbac-proxy-sar-config\""
Jul 10 18:52:13.000908 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.000834    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:13.005728 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.005711    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"]
Jul 10 18:52:13.100937 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.100903    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:13.161543 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.161507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66931a06-1aaf-4d64-b939-bf93a042e664-openshift-service-ca-bundle\") pod \"sequence-graph-34ebc-68f4d685-tf2f7\" (UID: \"66931a06-1aaf-4d64-b939-bf93a042e664\") " pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:13.161705 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.161603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66931a06-1aaf-4d64-b939-bf93a042e664-proxy-tls\") pod \"sequence-graph-34ebc-68f4d685-tf2f7\" (UID: \"66931a06-1aaf-4d64-b939-bf93a042e664\") " pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:13.262554 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.262465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66931a06-1aaf-4d64-b939-bf93a042e664-openshift-service-ca-bundle\") pod \"sequence-graph-34ebc-68f4d685-tf2f7\" (UID: \"66931a06-1aaf-4d64-b939-bf93a042e664\") " pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:13.262687 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.262557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66931a06-1aaf-4d64-b939-bf93a042e664-proxy-tls\") pod \"sequence-graph-34ebc-68f4d685-tf2f7\" (UID: \"66931a06-1aaf-4d64-b939-bf93a042e664\") " pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:13.263170 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.263141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66931a06-1aaf-4d64-b939-bf93a042e664-openshift-service-ca-bundle\") pod \"sequence-graph-34ebc-68f4d685-tf2f7\" (UID: \"66931a06-1aaf-4d64-b939-bf93a042e664\") " pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:13.265024 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.264996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66931a06-1aaf-4d64-b939-bf93a042e664-proxy-tls\") pod \"sequence-graph-34ebc-68f4d685-tf2f7\" (UID: \"66931a06-1aaf-4d64-b939-bf93a042e664\") " pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:13.308368 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.308331    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:13.438371 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:13.438347    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"]
Jul 10 18:52:13.440935 ip-10-0-129-73 kubenswrapper[2565]: W0710 18:52:13.440898    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66931a06_1aaf_4d64_b939_bf93a042e664.slice/crio-7f435e5d5feb35264d4510f6e68048ca1eed8623c4b7646448e76e4ad0f5992a WatchSource:0}: Error finding container 7f435e5d5feb35264d4510f6e68048ca1eed8623c4b7646448e76e4ad0f5992a: Status 404 returned error can't find the container with id 7f435e5d5feb35264d4510f6e68048ca1eed8623c4b7646448e76e4ad0f5992a
Jul 10 18:52:14.444289 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:14.444251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" event={"ID":"66931a06-1aaf-4d64-b939-bf93a042e664","Type":"ContainerStarted","Data":"16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b"}
Jul 10 18:52:14.444699 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:14.444311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" event={"ID":"66931a06-1aaf-4d64-b939-bf93a042e664","Type":"ContainerStarted","Data":"7f435e5d5feb35264d4510f6e68048ca1eed8623c4b7646448e76e4ad0f5992a"}
Jul 10 18:52:14.444699 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:14.444337    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:14.459941 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:14.459844    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" podStartSLOduration=2.459823981 podStartE2EDuration="2.459823981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:52:14.45951468 +0000 UTC m=+715.757856981" watchObservedRunningTime="2026-07-10 18:52:14.459823981 +0000 UTC m=+715.758166050"
Jul 10 18:52:20.454905 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:20.454878    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 18:52:49.280281 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:49.280244    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:51.817902 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:51.817866    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:52:51.874409 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:52:51.874372    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:53:11.622432 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:53:11.622398    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:54:58.212233 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:54:58.212195    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 18:55:19.164635 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:55:19.164602    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:55:19.165266 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:55:19.165249    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 18:59:41.469185 ip-10-0-129-73 kubenswrapper[2565]: I0710 18:59:41.469103    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:01.811713 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:01.811678    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"]
Jul 10 19:00:01.812302 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:01.811993    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08" containerID="cri-o://65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b" gracePeriod=30
Jul 10 19:00:01.835823 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:01.835760    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:02.044165 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.044135    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"]
Jul 10 19:00:02.047638 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.047616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.051977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.051928    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-48fe0-predictor-serving-cert\""
Jul 10 19:00:02.052107 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.052051    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-48fe0-kube-rbac-proxy-sar-config\""
Jul 10 19:00:02.060906 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.060883    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"]
Jul 10 19:00:02.075672 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.075595    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"]
Jul 10 19:00:02.075858 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.075835    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" containerID="cri-o://f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314" gracePeriod=30
Jul 10 19:00:02.075966 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.075926    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kube-rbac-proxy" containerID="cri-o://15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1" gracePeriod=30
Jul 10 19:00:02.129335 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.129298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.129535 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.129348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xb2\" (UniqueName: \"kubernetes.io/projected/6508d68c-ad79-4a6c-a298-b62e4493cc33-kube-api-access-c4xb2\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.129535 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.129464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6508d68c-ad79-4a6c-a298-b62e4493cc33-success-200-isvc-48fe0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.135535 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.135507    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"]
Jul 10 19:00:02.139058 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.139041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.141790 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.141769    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-48fe0-kube-rbac-proxy-sar-config\""
Jul 10 19:00:02.141924 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.141804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-48fe0-predictor-serving-cert\""
Jul 10 19:00:02.148646 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.148624    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"]
Jul 10 19:00:02.209465 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.209436    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"]
Jul 10 19:00:02.209666 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.209639    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" containerID="cri-o://671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f" gracePeriod=30
Jul 10 19:00:02.209791 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.209685    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kube-rbac-proxy" containerID="cri-o://e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33" gracePeriod=30
Jul 10 19:00:02.230176 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.230150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6508d68c-ad79-4a6c-a298-b62e4493cc33-success-200-isvc-48fe0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.230301 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.230199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/382b626a-a7cd-4cd7-a9ba-df3a8638f152-proxy-tls\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.230301 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.230237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.230409 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:02.230319    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-serving-cert: secret "success-200-isvc-48fe0-predictor-serving-cert" not found
Jul 10 19:00:02.230409 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.230328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjv4n\" (UniqueName: \"kubernetes.io/projected/382b626a-a7cd-4cd7-a9ba-df3a8638f152-kube-api-access-rjv4n\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.230409 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:02.230374    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls podName:6508d68c-ad79-4a6c-a298-b62e4493cc33 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:02.730352033 +0000 UTC m=+1184.028694094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls") pod "success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" (UID: "6508d68c-ad79-4a6c-a298-b62e4493cc33") : secret "success-200-isvc-48fe0-predictor-serving-cert" not found
Jul 10 19:00:02.230569 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.230495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xb2\" (UniqueName: \"kubernetes.io/projected/6508d68c-ad79-4a6c-a298-b62e4493cc33-kube-api-access-c4xb2\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.230626 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.230577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/382b626a-a7cd-4cd7-a9ba-df3a8638f152-error-404-isvc-48fe0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.230923 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.230901    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6508d68c-ad79-4a6c-a298-b62e4493cc33-success-200-isvc-48fe0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.244731 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.244709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xb2\" (UniqueName: \"kubernetes.io/projected/6508d68c-ad79-4a6c-a298-b62e4493cc33-kube-api-access-c4xb2\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.256270 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.256234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:02.331601 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.331571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/382b626a-a7cd-4cd7-a9ba-df3a8638f152-proxy-tls\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.331717 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.331625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjv4n\" (UniqueName: \"kubernetes.io/projected/382b626a-a7cd-4cd7-a9ba-df3a8638f152-kube-api-access-rjv4n\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.331717 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.331669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/382b626a-a7cd-4cd7-a9ba-df3a8638f152-error-404-isvc-48fe0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.332331 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.332309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/382b626a-a7cd-4cd7-a9ba-df3a8638f152-error-404-isvc-48fe0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.334485 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.334457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/382b626a-a7cd-4cd7-a9ba-df3a8638f152-proxy-tls\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.342242 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.342219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjv4n\" (UniqueName: \"kubernetes.io/projected/382b626a-a7cd-4cd7-a9ba-df3a8638f152-kube-api-access-rjv4n\") pod \"error-404-isvc-48fe0-predictor-87d87457b-zm2hn\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") " pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.353498 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.353467    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:02.387242 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.387214    2565 generic.go:356] "Generic (PLEG): container finished" podID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerID="e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33" exitCode=2
Jul 10 19:00:02.387411 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.387297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" event={"ID":"e720215f-d2fa-456b-9df8-40781dba0c7a","Type":"ContainerDied","Data":"e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33"}
Jul 10 19:00:02.388985 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.388942    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerID="15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1" exitCode=2
Jul 10 19:00:02.389095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.389018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" event={"ID":"a3c1737f-3369-4d97-a7b4-e021e0d034bd","Type":"ContainerDied","Data":"15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1"}
Jul 10 19:00:02.450228 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.450197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:02.454051 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.454018    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:02.591075 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.590996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"]
Jul 10 19:00:02.594427 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:00:02.594396    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382b626a_a7cd_4cd7_a9ba_df3a8638f152.slice/crio-d9161ac5611e22af94b5f441a9162e592d4b4caba2e31ab185eec787e81ac271 WatchSource:0}: Error finding container d9161ac5611e22af94b5f441a9162e592d4b4caba2e31ab185eec787e81ac271: Status 404 returned error can't find the container with id d9161ac5611e22af94b5f441a9162e592d4b4caba2e31ab185eec787e81ac271
Jul 10 19:00:02.736083 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.736053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.738419 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.738399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls\") pod \"success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") " pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:02.743941 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.743918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:02.959224 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:02.959186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:03.098941 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.098916    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"]
Jul 10 19:00:03.101485 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:00:03.101459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6508d68c_ad79_4a6c_a298_b62e4493cc33.slice/crio-3d795cefacce5d3b453160d26c17e5883112c1d8527ba971bbee571c651fc46f WatchSource:0}: Error finding container 3d795cefacce5d3b453160d26c17e5883112c1d8527ba971bbee571c651fc46f: Status 404 returned error can't find the container with id 3d795cefacce5d3b453160d26c17e5883112c1d8527ba971bbee571c651fc46f
Jul 10 19:00:03.397355 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.397318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" event={"ID":"6508d68c-ad79-4a6c-a298-b62e4493cc33","Type":"ContainerStarted","Data":"6e2c4a70cbef4d6dd8ef800ac061c95fe2b379582e427038cea9bed0484c53e2"}
Jul 10 19:00:03.397558 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.397400    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:03.397558 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.397530    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:03.397705 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.397584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" event={"ID":"6508d68c-ad79-4a6c-a298-b62e4493cc33","Type":"ContainerStarted","Data":"c5f8864a40c61bcba23be5532aafa5bec188e363012aca67baa2db0082cae758"}
Jul 10 19:00:03.397705 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.397636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" event={"ID":"6508d68c-ad79-4a6c-a298-b62e4493cc33","Type":"ContainerStarted","Data":"3d795cefacce5d3b453160d26c17e5883112c1d8527ba971bbee571c651fc46f"}
Jul 10 19:00:03.399202 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.399164    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:00:03.399313 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.399214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" event={"ID":"382b626a-a7cd-4cd7-a9ba-df3a8638f152","Type":"ContainerStarted","Data":"2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4"}
Jul 10 19:00:03.399313 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.399266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" event={"ID":"382b626a-a7cd-4cd7-a9ba-df3a8638f152","Type":"ContainerStarted","Data":"19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a"}
Jul 10 19:00:03.399313 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.399298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:03.399453 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.399327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" event={"ID":"382b626a-a7cd-4cd7-a9ba-df3a8638f152","Type":"ContainerStarted","Data":"d9161ac5611e22af94b5f441a9162e592d4b4caba2e31ab185eec787e81ac271"}
Jul 10 19:00:03.399453 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.399356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:03.400309 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.400285    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:00:03.414377 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.414316    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podStartSLOduration=1.41430058 podStartE2EDuration="1.41430058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:03.41408922 +0000 UTC m=+1184.712431290" watchObservedRunningTime="2026-07-10 19:00:03.41430058 +0000 UTC m=+1184.712642650"
Jul 10 19:00:03.432326 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.432259    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podStartSLOduration=1.432240075 podStartE2EDuration="1.432240075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:03.431797542 +0000 UTC m=+1184.730139625" watchObservedRunningTime="2026-07-10 19:00:03.432240075 +0000 UTC m=+1184.730582145"
Jul 10 19:00:03.988780 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.988735    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 10 19:00:03.994190 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:03.994161    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 19:00:04.330658 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:04.330563    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:04.404175 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:04.404120    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:00:04.404175 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:04.404161    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:00:04.992606 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:04.992557    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 10 19:00:04.998007 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:04.997942    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 19:00:05.983726 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:05.983696    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 19:00:06.015893 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.015865    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 19:00:06.064163 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.064134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn5rp\" (UniqueName: \"kubernetes.io/projected/e720215f-d2fa-456b-9df8-40781dba0c7a-kube-api-access-xn5rp\") pod \"e720215f-d2fa-456b-9df8-40781dba0c7a\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") "
Jul 10 19:00:06.064328 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.064183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8nnc\" (UniqueName: \"kubernetes.io/projected/a3c1737f-3369-4d97-a7b4-e021e0d034bd-kube-api-access-c8nnc\") pod \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") "
Jul 10 19:00:06.064328 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.064222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c1737f-3369-4d97-a7b4-e021e0d034bd-success-200-isvc-d6b08-kube-rbac-proxy-sar-config\") pod \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") "
Jul 10 19:00:06.064328 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.064271    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e720215f-d2fa-456b-9df8-40781dba0c7a-proxy-tls\") pod \"e720215f-d2fa-456b-9df8-40781dba0c7a\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") "
Jul 10 19:00:06.064328 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.064324    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e720215f-d2fa-456b-9df8-40781dba0c7a-error-404-isvc-d6b08-kube-rbac-proxy-sar-config\") pod \"e720215f-d2fa-456b-9df8-40781dba0c7a\" (UID: \"e720215f-d2fa-456b-9df8-40781dba0c7a\") "
Jul 10 19:00:06.064469 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.064356    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c1737f-3369-4d97-a7b4-e021e0d034bd-proxy-tls\") pod \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\" (UID: \"a3c1737f-3369-4d97-a7b4-e021e0d034bd\") "
Jul 10 19:00:06.064655 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.064626    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c1737f-3369-4d97-a7b4-e021e0d034bd-success-200-isvc-d6b08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-d6b08-kube-rbac-proxy-sar-config") pod "a3c1737f-3369-4d97-a7b4-e021e0d034bd" (UID: "a3c1737f-3369-4d97-a7b4-e021e0d034bd"). InnerVolumeSpecName "success-200-isvc-d6b08-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:06.064792 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.064721    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e720215f-d2fa-456b-9df8-40781dba0c7a-error-404-isvc-d6b08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-d6b08-kube-rbac-proxy-sar-config") pod "e720215f-d2fa-456b-9df8-40781dba0c7a" (UID: "e720215f-d2fa-456b-9df8-40781dba0c7a"). InnerVolumeSpecName "error-404-isvc-d6b08-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:06.066537 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.066507    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c1737f-3369-4d97-a7b4-e021e0d034bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3c1737f-3369-4d97-a7b4-e021e0d034bd" (UID: "a3c1737f-3369-4d97-a7b4-e021e0d034bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:06.066633 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.066510    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e720215f-d2fa-456b-9df8-40781dba0c7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e720215f-d2fa-456b-9df8-40781dba0c7a" (UID: "e720215f-d2fa-456b-9df8-40781dba0c7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:06.066633 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.066564    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e720215f-d2fa-456b-9df8-40781dba0c7a-kube-api-access-xn5rp" (OuterVolumeSpecName: "kube-api-access-xn5rp") pod "e720215f-d2fa-456b-9df8-40781dba0c7a" (UID: "e720215f-d2fa-456b-9df8-40781dba0c7a"). InnerVolumeSpecName "kube-api-access-xn5rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:06.066633 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.066592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c1737f-3369-4d97-a7b4-e021e0d034bd-kube-api-access-c8nnc" (OuterVolumeSpecName: "kube-api-access-c8nnc") pod "a3c1737f-3369-4d97-a7b4-e021e0d034bd" (UID: "a3c1737f-3369-4d97-a7b4-e021e0d034bd"). InnerVolumeSpecName "kube-api-access-c8nnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:06.165788 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.165752    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c1737f-3369-4d97-a7b4-e021e0d034bd-success-200-isvc-d6b08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:06.165788 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.165781    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e720215f-d2fa-456b-9df8-40781dba0c7a-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:06.165788 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.165793    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-d6b08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e720215f-d2fa-456b-9df8-40781dba0c7a-error-404-isvc-d6b08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:06.166049 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.165802    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c1737f-3369-4d97-a7b4-e021e0d034bd-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:06.166049 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.165812    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5rp\" (UniqueName: \"kubernetes.io/projected/e720215f-d2fa-456b-9df8-40781dba0c7a-kube-api-access-xn5rp\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:06.166049 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.165822    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8nnc\" (UniqueName: \"kubernetes.io/projected/a3c1737f-3369-4d97-a7b4-e021e0d034bd-kube-api-access-c8nnc\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:06.413756 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.413716    2565 generic.go:356] "Generic (PLEG): container finished" podID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerID="671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f" exitCode=0
Jul 10 19:00:06.413986 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.413798    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"
Jul 10 19:00:06.413986 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.413832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" event={"ID":"e720215f-d2fa-456b-9df8-40781dba0c7a","Type":"ContainerDied","Data":"671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f"}
Jul 10 19:00:06.413986 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.413899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn" event={"ID":"e720215f-d2fa-456b-9df8-40781dba0c7a","Type":"ContainerDied","Data":"e88b2f4b67c7a0cadffc9833e7873f3a2789095f0d5d1741de14301ed46db37f"}
Jul 10 19:00:06.413986 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.413941    2565 scope.go:117] "RemoveContainer" containerID="e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33"
Jul 10 19:00:06.415873 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.415842    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerID="f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314" exitCode=0
Jul 10 19:00:06.416030 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.415920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" event={"ID":"a3c1737f-3369-4d97-a7b4-e021e0d034bd","Type":"ContainerDied","Data":"f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314"}
Jul 10 19:00:06.416030 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.415921    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"
Jul 10 19:00:06.416156 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.416063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf" event={"ID":"a3c1737f-3369-4d97-a7b4-e021e0d034bd","Type":"ContainerDied","Data":"78aed6428a6c7ca332df5cc6a169058bb1acb51c6478b6ab130c83d2eb96fa31"}
Jul 10 19:00:06.424665 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.424640    2565 scope.go:117] "RemoveContainer" containerID="671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f"
Jul 10 19:00:06.433916 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.433872    2565 scope.go:117] "RemoveContainer" containerID="e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33"
Jul 10 19:00:06.434230 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:06.434208    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33\": container with ID starting with e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33 not found: ID does not exist" containerID="e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33"
Jul 10 19:00:06.434325 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.434238    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33"} err="failed to get container status \"e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33\": rpc error: code = NotFound desc = could not find container \"e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33\": container with ID starting with e5d01e1095196ef57459bf60c6b065b8a8219c193d5a0259213a5263a0ff1d33 not found: ID does not exist"
Jul 10 19:00:06.434325 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.434255    2565 scope.go:117] "RemoveContainer" containerID="671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f"
Jul 10 19:00:06.434504 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:06.434489    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f\": container with ID starting with 671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f not found: ID does not exist" containerID="671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f"
Jul 10 19:00:06.434541 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.434510    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f"} err="failed to get container status \"671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f\": rpc error: code = NotFound desc = could not find container \"671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f\": container with ID starting with 671137e19baf63b255a04eedecbdb4b860fa1f3868da32b873f514634339973f not found: ID does not exist"
Jul 10 19:00:06.434541 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.434524    2565 scope.go:117] "RemoveContainer" containerID="15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1"
Jul 10 19:00:06.441900 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.441861    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"]
Jul 10 19:00:06.442573 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.442556    2565 scope.go:117] "RemoveContainer" containerID="f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314"
Jul 10 19:00:06.446182 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.446160    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6b08-predictor-6878886558-5kjcn"]
Jul 10 19:00:06.450839 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.450812    2565 scope.go:117] "RemoveContainer" containerID="15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1"
Jul 10 19:00:06.451264 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:06.451243    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1\": container with ID starting with 15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1 not found: ID does not exist" containerID="15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1"
Jul 10 19:00:06.451324 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.451272    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1"} err="failed to get container status \"15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1\": rpc error: code = NotFound desc = could not find container \"15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1\": container with ID starting with 15f7204db472bee69c57b5be4bbec6115e000f3b7d6e4bcc3284ce86fa473be1 not found: ID does not exist"
Jul 10 19:00:06.451324 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.451297    2565 scope.go:117] "RemoveContainer" containerID="f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314"
Jul 10 19:00:06.451566 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:06.451549    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314\": container with ID starting with f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314 not found: ID does not exist" containerID="f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314"
Jul 10 19:00:06.451616 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.451569    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314"} err="failed to get container status \"f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314\": rpc error: code = NotFound desc = could not find container \"f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314\": container with ID starting with f899b30e1b1a10d8dd7223e0c4b67e4f3397a230903bb1ff51d64d1c1026b314 not found: ID does not exist"
Jul 10 19:00:06.457689 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.457655    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"]
Jul 10 19:00:06.467603 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:06.467571    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6b08-predictor-7ff55c75f7-t5xsf"]
Jul 10 19:00:07.273544 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:07.273504    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" path="/var/lib/kubelet/pods/a3c1737f-3369-4d97-a7b4-e021e0d034bd/volumes"
Jul 10 19:00:07.274112 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:07.274091    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" path="/var/lib/kubelet/pods/e720215f-d2fa-456b-9df8-40781dba0c7a/volumes"
Jul 10 19:00:09.331323 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:09.331284    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:09.408502 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:09.408473    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:09.409023 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:09.408991    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:00:09.409240 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:09.409221    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:00:09.409657 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:09.409634    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:00:14.331105 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:14.331067    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:14.331581 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:14.331175    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 19:00:19.192765 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:19.192735    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:00:19.193585 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:19.193563    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:00:19.331331 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:19.331294    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:19.409481 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:19.409447    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:00:19.409648 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:19.409594    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:00:24.331040 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:24.330996    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:27.733291 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.733259    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"]
Jul 10 19:00:27.733657 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.733452    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc" containerID="cri-o://16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b" gracePeriod=30
Jul 10 19:00:27.758635 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.758599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:27.859855 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.859821    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"]
Jul 10 19:00:27.860134 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.860105    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container" containerID="cri-o://3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1" gracePeriod=30
Jul 10 19:00:27.860229 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.860163    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kube-rbac-proxy" containerID="cri-o://86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814" gracePeriod=30
Jul 10 19:00:27.891180 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.891153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:27.908678 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.908654    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"]
Jul 10 19:00:27.909041 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909028    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kube-rbac-proxy"
Jul 10 19:00:27.909095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909043    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kube-rbac-proxy"
Jul 10 19:00:27.909095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909056    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container"
Jul 10 19:00:27.909095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909061    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container"
Jul 10 19:00:27.909095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909068    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kube-rbac-proxy"
Jul 10 19:00:27.909095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909074    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kube-rbac-proxy"
Jul 10 19:00:27.909095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909086    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container"
Jul 10 19:00:27.909095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909093    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container"
Jul 10 19:00:27.909300 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909143    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kserve-container"
Jul 10 19:00:27.909300 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909152    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kube-rbac-proxy"
Jul 10 19:00:27.909300 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909161    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c1737f-3369-4d97-a7b4-e021e0d034bd" containerName="kube-rbac-proxy"
Jul 10 19:00:27.909300 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.909169    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e720215f-d2fa-456b-9df8-40781dba0c7a" containerName="kserve-container"
Jul 10 19:00:27.913277 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.913260    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:27.917897 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.917872    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9304d-predictor-serving-cert\""
Jul 10 19:00:27.918219 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.918203    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9304d-kube-rbac-proxy-sar-config\""
Jul 10 19:00:27.932148 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.932122    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"]
Jul 10 19:00:27.988529 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.988459    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:27.994599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.994573    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"]
Jul 10 19:00:27.994852 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.994822    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container" containerID="cri-o://7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36" gracePeriod=30
Jul 10 19:00:27.994985 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:27.994854    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kube-rbac-proxy" containerID="cri-o://778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090" gracePeriod=30
Jul 10 19:00:28.040611 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.040578    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"]
Jul 10 19:00:28.044252 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.044234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.046989 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.046945    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9304d-kube-rbac-proxy-sar-config\""
Jul 10 19:00:28.047115 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.046944    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9304d-predictor-serving-cert\""
Jul 10 19:00:28.050016 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.049992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bc5a568-590c-4578-b197-840c4a3818fa-success-200-isvc-9304d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.050334 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.050235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bc5a568-590c-4578-b197-840c4a3818fa-proxy-tls\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.050434 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.050335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9t64\" (UniqueName: \"kubernetes.io/projected/9bc5a568-590c-4578-b197-840c4a3818fa-kube-api-access-k9t64\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.055669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.055645    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"]
Jul 10 19:00:28.084095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.084057    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:28.151681 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.151644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bc5a568-590c-4578-b197-840c4a3818fa-proxy-tls\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.151857 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.151695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-proxy-tls\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.151857 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.151818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9t64\" (UniqueName: \"kubernetes.io/projected/9bc5a568-590c-4578-b197-840c4a3818fa-kube-api-access-k9t64\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.151985 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.151865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bc5a568-590c-4578-b197-840c4a3818fa-success-200-isvc-9304d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.151985 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.151898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-error-404-isvc-9304d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.152107 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.151984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn84\" (UniqueName: \"kubernetes.io/projected/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-kube-api-access-dpn84\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.152575 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.152548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bc5a568-590c-4578-b197-840c4a3818fa-success-200-isvc-9304d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.154533 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.154512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bc5a568-590c-4578-b197-840c4a3818fa-proxy-tls\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.163403 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.163378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9t64\" (UniqueName: \"kubernetes.io/projected/9bc5a568-590c-4578-b197-840c4a3818fa-kube-api-access-k9t64\") pod \"success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") " pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.204330 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.204287    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:00:28.223267 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.223249    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.257027 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.252528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn84\" (UniqueName: \"kubernetes.io/projected/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-kube-api-access-dpn84\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.257027 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.252593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-proxy-tls\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.257027 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.252665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-error-404-isvc-9304d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.257027 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.253393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-error-404-isvc-9304d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.257027 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.255477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-proxy-tls\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.262770 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.262744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn84\" (UniqueName: \"kubernetes.io/projected/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-kube-api-access-dpn84\") pod \"error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") " pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.357161 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.357128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:28.358719 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.358697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"]
Jul 10 19:00:28.360765 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:00:28.360736    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc5a568_590c_4578_b197_840c4a3818fa.slice/crio-f817e4e1b91eafecebe0e928ab6cf7e22d747973fd2014df930640fecb7f5e92 WatchSource:0}: Error finding container f817e4e1b91eafecebe0e928ab6cf7e22d747973fd2014df930640fecb7f5e92: Status 404 returned error can't find the container with id f817e4e1b91eafecebe0e928ab6cf7e22d747973fd2014df930640fecb7f5e92
Jul 10 19:00:28.508007 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.507926    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"]
Jul 10 19:00:28.511076 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:00:28.511045    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee2f1c9_66ce_43fd_9e6f_019f81023f1d.slice/crio-c1d48e2ffdd5fa875a3f2c3653fc0e427a168bbfc78504bf35a95ed9863fb3a3 WatchSource:0}: Error finding container c1d48e2ffdd5fa875a3f2c3653fc0e427a168bbfc78504bf35a95ed9863fb3a3: Status 404 returned error can't find the container with id c1d48e2ffdd5fa875a3f2c3653fc0e427a168bbfc78504bf35a95ed9863fb3a3
Jul 10 19:00:28.513757 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.513733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" event={"ID":"9bc5a568-590c-4578-b197-840c4a3818fa","Type":"ContainerStarted","Data":"c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77"}
Jul 10 19:00:28.513856 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.513797    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" event={"ID":"9bc5a568-590c-4578-b197-840c4a3818fa","Type":"ContainerStarted","Data":"f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a"}
Jul 10 19:00:28.513856 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.513833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" event={"ID":"9bc5a568-590c-4578-b197-840c4a3818fa","Type":"ContainerStarted","Data":"f817e4e1b91eafecebe0e928ab6cf7e22d747973fd2014df930640fecb7f5e92"}
Jul 10 19:00:28.513989 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.513868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:28.515499 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.515478    2565 generic.go:356] "Generic (PLEG): container finished" podID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerID="86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814" exitCode=2
Jul 10 19:00:28.515585 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.515551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" event={"ID":"a1970204-9768-4dec-9cbc-3e47d4620f18","Type":"ContainerDied","Data":"86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814"}
Jul 10 19:00:28.517329 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.517276    2565 generic.go:356] "Generic (PLEG): container finished" podID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerID="778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090" exitCode=2
Jul 10 19:00:28.517329 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:28.517315    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" event={"ID":"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8","Type":"ContainerDied","Data":"778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090"}
Jul 10 19:00:29.330779 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.330743    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:29.409377 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.409341    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:00:29.409771 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.409749    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:00:29.522912 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.522877    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" event={"ID":"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d","Type":"ContainerStarted","Data":"1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a"}
Jul 10 19:00:29.523210 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.523048    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:29.523283 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.523245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:29.523344 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.523308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" event={"ID":"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d","Type":"ContainerStarted","Data":"9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277"}
Jul 10 19:00:29.523393 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.523347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" event={"ID":"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d","Type":"ContainerStarted","Data":"c1d48e2ffdd5fa875a3f2c3653fc0e427a168bbfc78504bf35a95ed9863fb3a3"}
Jul 10 19:00:29.523393 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.523378    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:29.524203 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.524175    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:00:29.524332 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.524273    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:00:29.543700 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.543654    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podStartSLOduration=1.5436391010000001 podStartE2EDuration="1.543639101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:29.542235944 +0000 UTC m=+1210.840578012" watchObservedRunningTime="2026-07-10 19:00:29.543639101 +0000 UTC m=+1210.841981170"
Jul 10 19:00:29.544256 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:29.544230    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podStartSLOduration=2.544222911 podStartE2EDuration="2.544222911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:28.53824711 +0000 UTC m=+1209.836589198" watchObservedRunningTime="2026-07-10 19:00:29.544222911 +0000 UTC m=+1210.842564981"
Jul 10 19:00:30.168360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:30.168321    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 10 19:00:30.168507 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:30.168321    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 10 19:00:30.174653 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:30.174625    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 19:00:30.174783 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:30.174623    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 19:00:30.461295 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:30.461204    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:30.527725 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:30.527682    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:00:30.527918 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:30.527782    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:00:31.315453 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.315424    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 19:00:31.381858 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.381822    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1970204-9768-4dec-9cbc-3e47d4620f18-success-200-isvc-34ebc-kube-rbac-proxy-sar-config\") pod \"a1970204-9768-4dec-9cbc-3e47d4620f18\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") "
Jul 10 19:00:31.381858 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.381862    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1970204-9768-4dec-9cbc-3e47d4620f18-proxy-tls\") pod \"a1970204-9768-4dec-9cbc-3e47d4620f18\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") "
Jul 10 19:00:31.382089 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.381884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgxwc\" (UniqueName: \"kubernetes.io/projected/a1970204-9768-4dec-9cbc-3e47d4620f18-kube-api-access-sgxwc\") pod \"a1970204-9768-4dec-9cbc-3e47d4620f18\" (UID: \"a1970204-9768-4dec-9cbc-3e47d4620f18\") "
Jul 10 19:00:31.382221 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.382196    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1970204-9768-4dec-9cbc-3e47d4620f18-success-200-isvc-34ebc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-34ebc-kube-rbac-proxy-sar-config") pod "a1970204-9768-4dec-9cbc-3e47d4620f18" (UID: "a1970204-9768-4dec-9cbc-3e47d4620f18"). InnerVolumeSpecName "success-200-isvc-34ebc-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:31.384180 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.384147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1970204-9768-4dec-9cbc-3e47d4620f18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1970204-9768-4dec-9cbc-3e47d4620f18" (UID: "a1970204-9768-4dec-9cbc-3e47d4620f18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:31.384180 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.384163    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1970204-9768-4dec-9cbc-3e47d4620f18-kube-api-access-sgxwc" (OuterVolumeSpecName: "kube-api-access-sgxwc") pod "a1970204-9768-4dec-9cbc-3e47d4620f18" (UID: "a1970204-9768-4dec-9cbc-3e47d4620f18"). InnerVolumeSpecName "kube-api-access-sgxwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:31.482802 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.482759    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1970204-9768-4dec-9cbc-3e47d4620f18-success-200-isvc-34ebc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:31.482802 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.482795    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1970204-9768-4dec-9cbc-3e47d4620f18-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:31.483243 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.482820    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgxwc\" (UniqueName: \"kubernetes.io/projected/a1970204-9768-4dec-9cbc-3e47d4620f18-kube-api-access-sgxwc\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:31.533413 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.533378    2565 generic.go:356] "Generic (PLEG): container finished" podID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerID="3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1" exitCode=0
Jul 10 19:00:31.533603 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.533435    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"
Jul 10 19:00:31.533603 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.533468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" event={"ID":"a1970204-9768-4dec-9cbc-3e47d4620f18","Type":"ContainerDied","Data":"3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1"}
Jul 10 19:00:31.533603 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.533526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn" event={"ID":"a1970204-9768-4dec-9cbc-3e47d4620f18","Type":"ContainerDied","Data":"3553b60b2fe4de1dacf1c5537f27feb5d4677694e4761b2c1da6eac973be39a8"}
Jul 10 19:00:31.533603 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.533554    2565 scope.go:117] "RemoveContainer" containerID="86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814"
Jul 10 19:00:31.534134 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.534094    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:00:31.542719 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.542702    2565 scope.go:117] "RemoveContainer" containerID="3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1"
Jul 10 19:00:31.551918 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.551871    2565 scope.go:117] "RemoveContainer" containerID="86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814"
Jul 10 19:00:31.552389 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:31.552360    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814\": container with ID starting with 86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814 not found: ID does not exist" containerID="86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814"
Jul 10 19:00:31.552507 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.552400    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814"} err="failed to get container status \"86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814\": rpc error: code = NotFound desc = could not find container \"86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814\": container with ID starting with 86d10f8ea2341c85376c053c3bc592446b08c69a2d0a8d738f1fb784eae60814 not found: ID does not exist"
Jul 10 19:00:31.552507 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.552426    2565 scope.go:117] "RemoveContainer" containerID="3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1"
Jul 10 19:00:31.552742 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:31.552720    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1\": container with ID starting with 3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1 not found: ID does not exist" containerID="3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1"
Jul 10 19:00:31.552806 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.552754    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1"} err="failed to get container status \"3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1\": rpc error: code = NotFound desc = could not find container \"3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1\": container with ID starting with 3fbb1d2518ad3b982b0c413e4ee2b273680bf794ce71a00b9e1703edb2a352f1 not found: ID does not exist"
Jul 10 19:00:31.558529 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.558502    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"]
Jul 10 19:00:31.563496 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:31.563469    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-34ebc-predictor-86f8b97c9-5m7wn"]
Jul 10 19:00:32.041131 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.041106    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 19:00:32.191074 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.191046    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-proxy-tls\") pod \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") "
Jul 10 19:00:32.191245 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.191108    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-526wh\" (UniqueName: \"kubernetes.io/projected/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-kube-api-access-526wh\") pod \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") "
Jul 10 19:00:32.191245 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.191151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-error-404-isvc-34ebc-kube-rbac-proxy-sar-config\") pod \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\" (UID: \"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8\") "
Jul 10 19:00:32.191592 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.191565    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-error-404-isvc-34ebc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-34ebc-kube-rbac-proxy-sar-config") pod "f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" (UID: "f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8"). InnerVolumeSpecName "error-404-isvc-34ebc-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:32.193232 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.193203    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" (UID: "f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:32.193331 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.193312    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-kube-api-access-526wh" (OuterVolumeSpecName: "kube-api-access-526wh") pod "f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" (UID: "f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8"). InnerVolumeSpecName "kube-api-access-526wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:32.291790 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.291751    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:32.291790 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.291783    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-526wh\" (UniqueName: \"kubernetes.io/projected/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-kube-api-access-526wh\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:32.291790 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.291794    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-34ebc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8-error-404-isvc-34ebc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:32.477116 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.477093    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 19:00:32.539886 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.539854    2565 generic.go:356] "Generic (PLEG): container finished" podID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerID="7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36" exitCode=0
Jul 10 19:00:32.540309 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.539889    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" event={"ID":"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8","Type":"ContainerDied","Data":"7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36"}
Jul 10 19:00:32.540309 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.539923    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"
Jul 10 19:00:32.540309 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.539973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2" event={"ID":"f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8","Type":"ContainerDied","Data":"faa26ba7c1f25b575b64aa23d832d4174c10c0c08810ebecdd6881205e2f9722"}
Jul 10 19:00:32.540309 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.539996    2565 scope.go:117] "RemoveContainer" containerID="778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090"
Jul 10 19:00:32.541246 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.541224    2565 generic.go:356] "Generic (PLEG): container finished" podID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerID="65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b" exitCode=0
Jul 10 19:00:32.541344 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.541273    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"
Jul 10 19:00:32.541344 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.541313    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" event={"ID":"87a3d8b0-b196-4247-8d71-c6118f8a6a4f","Type":"ContainerDied","Data":"65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b"}
Jul 10 19:00:32.541608 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.541354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk" event={"ID":"87a3d8b0-b196-4247-8d71-c6118f8a6a4f","Type":"ContainerDied","Data":"f1c02d7e76c401b30c3b2d27c4aa9287d22d63129f5316193ccf0c9cbe8a3558"}
Jul 10 19:00:32.548659 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.548642    2565 scope.go:117] "RemoveContainer" containerID="7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36"
Jul 10 19:00:32.557247 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.557226    2565 scope.go:117] "RemoveContainer" containerID="778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090"
Jul 10 19:00:32.557522 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:32.557503    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090\": container with ID starting with 778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090 not found: ID does not exist" containerID="778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090"
Jul 10 19:00:32.557572 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.557534    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090"} err="failed to get container status \"778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090\": rpc error: code = NotFound desc = could not find container \"778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090\": container with ID starting with 778a0de4c8d9a30f3611b334ded8c2136ac479f5c1f1d08f1f78a5a8c3690090 not found: ID does not exist"
Jul 10 19:00:32.557572 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.557552    2565 scope.go:117] "RemoveContainer" containerID="7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36"
Jul 10 19:00:32.557809 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:32.557787    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36\": container with ID starting with 7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36 not found: ID does not exist" containerID="7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36"
Jul 10 19:00:32.557850 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.557815    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36"} err="failed to get container status \"7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36\": rpc error: code = NotFound desc = could not find container \"7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36\": container with ID starting with 7732b88f3e67f3fe007492e2511ef37a04b0ec17239a1893f937d6a539112f36 not found: ID does not exist"
Jul 10 19:00:32.557850 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.557831    2565 scope.go:117] "RemoveContainer" containerID="65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b"
Jul 10 19:00:32.566145 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.566118    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"]
Jul 10 19:00:32.566351 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.566315    2565 scope.go:117] "RemoveContainer" containerID="65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b"
Jul 10 19:00:32.566586 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:32.566568    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b\": container with ID starting with 65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b not found: ID does not exist" containerID="65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b"
Jul 10 19:00:32.566666 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.566591    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b"} err="failed to get container status \"65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b\": rpc error: code = NotFound desc = could not find container \"65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b\": container with ID starting with 65fa4d820caeae7b052e2d3d2c32d7f5984f63aad727ed101954708d64966f6b not found: ID does not exist"
Jul 10 19:00:32.567710 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.567687    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34ebc-predictor-59d677957b-wtgd2"]
Jul 10 19:00:32.593599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.593577    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-openshift-service-ca-bundle\") pod \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\" (UID: \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\") "
Jul 10 19:00:32.593699 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.593611    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-proxy-tls\") pod \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\" (UID: \"87a3d8b0-b196-4247-8d71-c6118f8a6a4f\") "
Jul 10 19:00:32.593996 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.593947    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "87a3d8b0-b196-4247-8d71-c6118f8a6a4f" (UID: "87a3d8b0-b196-4247-8d71-c6118f8a6a4f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:32.595712 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.595695    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87a3d8b0-b196-4247-8d71-c6118f8a6a4f" (UID: "87a3d8b0-b196-4247-8d71-c6118f8a6a4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:32.695120 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.695077    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:32.695120 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.695112    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a3d8b0-b196-4247-8d71-c6118f8a6a4f-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:32.863790 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.863753    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"]
Jul 10 19:00:32.868417 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:32.868386    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d6b08-56b6fff7d5-vhpwk"]
Jul 10 19:00:33.273590 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:33.273555    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" path="/var/lib/kubelet/pods/87a3d8b0-b196-4247-8d71-c6118f8a6a4f/volumes"
Jul 10 19:00:33.273927 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:33.273914    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" path="/var/lib/kubelet/pods/a1970204-9768-4dec-9cbc-3e47d4620f18/volumes"
Jul 10 19:00:33.274326 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:33.274313    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" path="/var/lib/kubelet/pods/f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8/volumes"
Jul 10 19:00:35.455219 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:35.455175    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:35.531634 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:35.531606    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:00:35.532128 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:35.532098    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:00:36.538260 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:36.538228    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:00:36.538700 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:36.538677    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:00:39.409189 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:39.409149    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:00:39.409645 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:39.409623    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:00:40.455447 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:40.455401    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:40.455932 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:40.455544    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 19:00:45.455837 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:45.455779    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:45.532128 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:45.532087    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:00:46.539337 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:46.539292    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:00:49.409314 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:49.409274    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:00:49.409706 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:49.409561    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:00:50.454127 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:50.454083    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:55.455328 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:55.455287    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:00:55.532313 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:55.532272    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:00:56.538927 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:56.538879    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:00:57.886082 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:57.886058    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 19:00:57.898936 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:57.898909    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66931a06-1aaf-4d64-b939-bf93a042e664-proxy-tls\") pod \"66931a06-1aaf-4d64-b939-bf93a042e664\" (UID: \"66931a06-1aaf-4d64-b939-bf93a042e664\") "
Jul 10 19:00:57.899104 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:57.898974    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66931a06-1aaf-4d64-b939-bf93a042e664-openshift-service-ca-bundle\") pod \"66931a06-1aaf-4d64-b939-bf93a042e664\" (UID: \"66931a06-1aaf-4d64-b939-bf93a042e664\") "
Jul 10 19:00:57.899298 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:57.899274    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66931a06-1aaf-4d64-b939-bf93a042e664-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "66931a06-1aaf-4d64-b939-bf93a042e664" (UID: "66931a06-1aaf-4d64-b939-bf93a042e664"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:57.901169 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:57.901142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66931a06-1aaf-4d64-b939-bf93a042e664-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66931a06-1aaf-4d64-b939-bf93a042e664" (UID: "66931a06-1aaf-4d64-b939-bf93a042e664"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:58.000133 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.000041    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66931a06-1aaf-4d64-b939-bf93a042e664-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:58.000133 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.000075    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66931a06-1aaf-4d64-b939-bf93a042e664-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:00:58.657883 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.657849    2565 generic.go:356] "Generic (PLEG): container finished" podID="66931a06-1aaf-4d64-b939-bf93a042e664" containerID="16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b" exitCode=0
Jul 10 19:00:58.658099 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.657893    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"
Jul 10 19:00:58.658099 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.657937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" event={"ID":"66931a06-1aaf-4d64-b939-bf93a042e664","Type":"ContainerDied","Data":"16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b"}
Jul 10 19:00:58.658099 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.658017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7" event={"ID":"66931a06-1aaf-4d64-b939-bf93a042e664","Type":"ContainerDied","Data":"7f435e5d5feb35264d4510f6e68048ca1eed8623c4b7646448e76e4ad0f5992a"}
Jul 10 19:00:58.658099 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.658049    2565 scope.go:117] "RemoveContainer" containerID="16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b"
Jul 10 19:00:58.666989 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.666948    2565 scope.go:117] "RemoveContainer" containerID="16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b"
Jul 10 19:00:58.667259 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:00:58.667243    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b\": container with ID starting with 16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b not found: ID does not exist" containerID="16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b"
Jul 10 19:00:58.667305 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.667269    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b"} err="failed to get container status \"16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b\": rpc error: code = NotFound desc = could not find container \"16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b\": container with ID starting with 16ab8349fa78897f92887c02760c8ce773da85b5b23da835eff8b406aa22eb1b not found: ID does not exist"
Jul 10 19:00:58.681002 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.680971    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"]
Jul 10 19:00:58.685528 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:58.685508    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-34ebc-68f4d685-tf2f7"]
Jul 10 19:00:59.273236 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:59.273203    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" path="/var/lib/kubelet/pods/66931a06-1aaf-4d64-b939-bf93a042e664/volumes"
Jul 10 19:00:59.410122 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:59.410097    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:00:59.410289 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:00:59.410227    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:01:00.827290 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:00.827254    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:05.532189 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:05.532146    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:01:06.538971 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:06.538914    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:01:12.068945 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.068865    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"]
Jul 10 19:01:12.069282 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069261    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container"
Jul 10 19:01:12.069282 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069272    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container"
Jul 10 19:01:12.069282 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069280    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kube-rbac-proxy"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069285    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kube-rbac-proxy"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069295    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069302    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069310    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kube-rbac-proxy"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069315    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kube-rbac-proxy"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069328    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069333    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069348    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069353    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container"
Jul 10 19:01:12.069396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069399    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a3d8b0-b196-4247-8d71-c6118f8a6a4f" containerName="switch-graph-d6b08"
Jul 10 19:01:12.069673 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069407    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kube-rbac-proxy"
Jul 10 19:01:12.069673 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069413    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ac9ffb-3d16-443b-a7d0-9d43a1e82dc8" containerName="kserve-container"
Jul 10 19:01:12.069673 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069419    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kube-rbac-proxy"
Jul 10 19:01:12.069673 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069427    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66931a06-1aaf-4d64-b939-bf93a042e664" containerName="sequence-graph-34ebc"
Jul 10 19:01:12.069673 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.069433    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1970204-9768-4dec-9cbc-3e47d4620f18" containerName="kserve-container"
Jul 10 19:01:12.073792 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.073774    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:12.076410 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.076388    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-48fe0-serving-cert\""
Jul 10 19:01:12.076680 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.076659    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-48fe0-kube-rbac-proxy-sar-config\""
Jul 10 19:01:12.077153 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.077133    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:12.081915 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.081894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"]
Jul 10 19:01:12.120941 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.120902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601bf731-2dff-4309-ac48-8aeb66d64977-openshift-service-ca-bundle\") pod \"ensemble-graph-48fe0-57cb8d9dbf-qqrpz\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") " pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:12.121115 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.120944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls\") pod \"ensemble-graph-48fe0-57cb8d9dbf-qqrpz\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") " pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:12.222018 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.221975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601bf731-2dff-4309-ac48-8aeb66d64977-openshift-service-ca-bundle\") pod \"ensemble-graph-48fe0-57cb8d9dbf-qqrpz\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") " pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:12.222165 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.222035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls\") pod \"ensemble-graph-48fe0-57cb8d9dbf-qqrpz\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") " pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:12.222165 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:12.222155    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-48fe0-serving-cert: secret "ensemble-graph-48fe0-serving-cert" not found
Jul 10 19:01:12.222254 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:12.222221    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls podName:601bf731-2dff-4309-ac48-8aeb66d64977 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:12.722204429 +0000 UTC m=+1254.020546476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls") pod "ensemble-graph-48fe0-57cb8d9dbf-qqrpz" (UID: "601bf731-2dff-4309-ac48-8aeb66d64977") : secret "ensemble-graph-48fe0-serving-cert" not found
Jul 10 19:01:12.222614 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.222593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601bf731-2dff-4309-ac48-8aeb66d64977-openshift-service-ca-bundle\") pod \"ensemble-graph-48fe0-57cb8d9dbf-qqrpz\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") " pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:12.372384 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.372292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:12.726728 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.726699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls\") pod \"ensemble-graph-48fe0-57cb8d9dbf-qqrpz\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") " pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:12.729267 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.729246    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls\") pod \"ensemble-graph-48fe0-57cb8d9dbf-qqrpz\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") " pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:12.984435 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:12.984352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:13.118759 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:13.118733    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"]
Jul 10 19:01:13.120643 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:01:13.120614    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601bf731_2dff_4309_ac48_8aeb66d64977.slice/crio-6f03fdeed89d08c184b0c654725161856281918fd1ccd165930d449e404ed9ad WatchSource:0}: Error finding container 6f03fdeed89d08c184b0c654725161856281918fd1ccd165930d449e404ed9ad: Status 404 returned error can't find the container with id 6f03fdeed89d08c184b0c654725161856281918fd1ccd165930d449e404ed9ad
Jul 10 19:01:13.723323 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:13.723285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" event={"ID":"601bf731-2dff-4309-ac48-8aeb66d64977","Type":"ContainerStarted","Data":"20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db"}
Jul 10 19:01:13.723520 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:13.723340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" event={"ID":"601bf731-2dff-4309-ac48-8aeb66d64977","Type":"ContainerStarted","Data":"6f03fdeed89d08c184b0c654725161856281918fd1ccd165930d449e404ed9ad"}
Jul 10 19:01:13.723520 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:13.723414    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:13.744131 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:13.744088    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" podStartSLOduration=1.744072675 podStartE2EDuration="1.744072675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:13.743046151 +0000 UTC m=+1255.041388220" watchObservedRunningTime="2026-07-10 19:01:13.744072675 +0000 UTC m=+1255.042414743"
Jul 10 19:01:15.532940 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:15.532913    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:01:16.539643 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:16.539609    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:01:19.733758 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:19.733731    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:22.133476 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.133442    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"]
Jul 10 19:01:22.133888 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.133621    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0" containerID="cri-o://20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db" gracePeriod=30
Jul 10 19:01:22.156013 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.155944    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:22.253568 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.253528    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"]
Jul 10 19:01:22.253839 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.253783    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container" containerID="cri-o://19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a" gracePeriod=30
Jul 10 19:01:22.253899 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.253830    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kube-rbac-proxy" containerID="cri-o://2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4" gracePeriod=30
Jul 10 19:01:22.278714 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.278682    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:22.295835 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.295795    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"]
Jul 10 19:01:22.300914 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.300894    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.303263 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.303239    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9b322-predictor-serving-cert\""
Jul 10 19:01:22.303402 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.303386    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9b322-kube-rbac-proxy-sar-config\""
Jul 10 19:01:22.309366 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.309336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"]
Jul 10 19:01:22.356682 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.356648    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"]
Jul 10 19:01:22.357563 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.357503    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container" containerID="cri-o://c5f8864a40c61bcba23be5532aafa5bec188e363012aca67baa2db0082cae758" gracePeriod=30
Jul 10 19:01:22.357789 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.357756    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kube-rbac-proxy" containerID="cri-o://6e2c4a70cbef4d6dd8ef800ac061c95fe2b379582e427038cea9bed0484c53e2" gracePeriod=30
Jul 10 19:01:22.362109 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.362087    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:22.410318 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.410231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be1b1971-a498-4212-8b72-12ccafffac4c-success-200-isvc-9b322-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.410318 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.410300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5ln\" (UniqueName: \"kubernetes.io/projected/be1b1971-a498-4212-8b72-12ccafffac4c-kube-api-access-zr5ln\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.410553 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.410329    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.412111 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.412088    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"]
Jul 10 19:01:22.415625 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.415610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.418344 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.418327    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9b322-kube-rbac-proxy-sar-config\""
Jul 10 19:01:22.418476 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.418343    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9b322-predictor-serving-cert\""
Jul 10 19:01:22.428878 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.428852    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"]
Jul 10 19:01:22.450774 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.450720    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:22.510781 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.510752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be1b1971-a498-4212-8b72-12ccafffac4c-success-200-isvc-9b322-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.510930 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.510797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.510930 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.510848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5ln\" (UniqueName: \"kubernetes.io/projected/be1b1971-a498-4212-8b72-12ccafffac4c-kube-api-access-zr5ln\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.510930 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.510867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.510930 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.510886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66ec695-62e3-4da4-92d6-4da6f9b69798-error-404-isvc-9b322-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.510930 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.510914    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvll\" (UniqueName: \"kubernetes.io/projected/e66ec695-62e3-4da4-92d6-4da6f9b69798-kube-api-access-5nvll\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.511175 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:22.511017    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-9b322-predictor-serving-cert: secret "success-200-isvc-9b322-predictor-serving-cert" not found
Jul 10 19:01:22.511175 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:22.511085    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls podName:be1b1971-a498-4212-8b72-12ccafffac4c nodeName:}" failed. No retries permitted until 2026-07-10 19:01:23.011062839 +0000 UTC m=+1264.309404885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls") pod "success-200-isvc-9b322-predictor-56ff68b897-sfxwq" (UID: "be1b1971-a498-4212-8b72-12ccafffac4c") : secret "success-200-isvc-9b322-predictor-serving-cert" not found
Jul 10 19:01:22.511587 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.511566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be1b1971-a498-4212-8b72-12ccafffac4c-success-200-isvc-9b322-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.520214 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.520191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5ln\" (UniqueName: \"kubernetes.io/projected/be1b1971-a498-4212-8b72-12ccafffac4c-kube-api-access-zr5ln\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:22.612031 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.612000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.612218 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.612049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66ec695-62e3-4da4-92d6-4da6f9b69798-error-404-isvc-9b322-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.612218 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.612072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvll\" (UniqueName: \"kubernetes.io/projected/e66ec695-62e3-4da4-92d6-4da6f9b69798-kube-api-access-5nvll\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.612218 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:22.612145    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9b322-predictor-serving-cert: secret "error-404-isvc-9b322-predictor-serving-cert" not found
Jul 10 19:01:22.612218 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:22.612206    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls podName:e66ec695-62e3-4da4-92d6-4da6f9b69798 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:23.112189281 +0000 UTC m=+1264.410531329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls") pod "error-404-isvc-9b322-predictor-657656b8f8-qjlvl" (UID: "e66ec695-62e3-4da4-92d6-4da6f9b69798") : secret "error-404-isvc-9b322-predictor-serving-cert" not found
Jul 10 19:01:22.612709 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.612684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66ec695-62e3-4da4-92d6-4da6f9b69798-error-404-isvc-9b322-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.623206 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.623182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvll\" (UniqueName: \"kubernetes.io/projected/e66ec695-62e3-4da4-92d6-4da6f9b69798-kube-api-access-5nvll\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:22.767076 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.767043    2565 generic.go:356] "Generic (PLEG): container finished" podID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerID="2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4" exitCode=2
Jul 10 19:01:22.767257 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.767117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" event={"ID":"382b626a-a7cd-4cd7-a9ba-df3a8638f152","Type":"ContainerDied","Data":"2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4"}
Jul 10 19:01:22.768866 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.768842    2565 generic.go:356] "Generic (PLEG): container finished" podID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerID="6e2c4a70cbef4d6dd8ef800ac061c95fe2b379582e427038cea9bed0484c53e2" exitCode=2
Jul 10 19:01:22.768992 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:22.768873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" event={"ID":"6508d68c-ad79-4a6c-a298-b62e4493cc33","Type":"ContainerDied","Data":"6e2c4a70cbef4d6dd8ef800ac061c95fe2b379582e427038cea9bed0484c53e2"}
Jul 10 19:01:23.015862 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.015816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:23.016074 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:23.015986    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-9b322-predictor-serving-cert: secret "success-200-isvc-9b322-predictor-serving-cert" not found
Jul 10 19:01:23.016074 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:23.016049    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls podName:be1b1971-a498-4212-8b72-12ccafffac4c nodeName:}" failed. No retries permitted until 2026-07-10 19:01:24.016032644 +0000 UTC m=+1265.314374691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls") pod "success-200-isvc-9b322-predictor-56ff68b897-sfxwq" (UID: "be1b1971-a498-4212-8b72-12ccafffac4c") : secret "success-200-isvc-9b322-predictor-serving-cert" not found
Jul 10 19:01:23.032358 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.032280    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:23.116829 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.116791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:23.119444 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.119421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls\") pod \"error-404-isvc-9b322-predictor-657656b8f8-qjlvl\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") " pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:23.130071 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.130036    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:23.325792 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.325705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:23.461563 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.461497    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"]
Jul 10 19:01:23.463718 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:01:23.463694    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66ec695_62e3_4da4_92d6_4da6f9b69798.slice/crio-1bb8a4f4cc9dce27b9257ece2880b5fa98e47737784577d8737e24ba483ac72a WatchSource:0}: Error finding container 1bb8a4f4cc9dce27b9257ece2880b5fa98e47737784577d8737e24ba483ac72a: Status 404 returned error can't find the container with id 1bb8a4f4cc9dce27b9257ece2880b5fa98e47737784577d8737e24ba483ac72a
Jul 10 19:01:23.774618 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.774584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" event={"ID":"e66ec695-62e3-4da4-92d6-4da6f9b69798","Type":"ContainerStarted","Data":"15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec"}
Jul 10 19:01:23.774795 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.774641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" event={"ID":"e66ec695-62e3-4da4-92d6-4da6f9b69798","Type":"ContainerStarted","Data":"d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d"}
Jul 10 19:01:23.774795 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.774662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" event={"ID":"e66ec695-62e3-4da4-92d6-4da6f9b69798","Type":"ContainerStarted","Data":"1bb8a4f4cc9dce27b9257ece2880b5fa98e47737784577d8737e24ba483ac72a"}
Jul 10 19:01:23.774795 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.774750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:23.797263 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:23.797211    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podStartSLOduration=1.797194632 podStartE2EDuration="1.797194632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:23.796365259 +0000 UTC m=+1265.094707327" watchObservedRunningTime="2026-07-10 19:01:23.797194632 +0000 UTC m=+1265.095536695"
Jul 10 19:01:24.024562 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.024514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:24.027130 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.027100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls\") pod \"success-200-isvc-9b322-predictor-56ff68b897-sfxwq\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") " pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:24.111974 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.111911    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:24.248197 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.248157    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"]
Jul 10 19:01:24.252838 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:01:24.252806    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1b1971_a498_4212_8b72_12ccafffac4c.slice/crio-65f5b8eb4210b5e6cea0c8c5238ab6d4ef42e3265c10fdba41c60221ab81b68a WatchSource:0}: Error finding container 65f5b8eb4210b5e6cea0c8c5238ab6d4ef42e3265c10fdba41c60221ab81b68a: Status 404 returned error can't find the container with id 65f5b8eb4210b5e6cea0c8c5238ab6d4ef42e3265c10fdba41c60221ab81b68a
Jul 10 19:01:24.405173 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.405135    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 10 19:01:24.405520 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.405134    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 10 19:01:24.733047 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.733002    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:24.781064 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.781025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" event={"ID":"be1b1971-a498-4212-8b72-12ccafffac4c","Type":"ContainerStarted","Data":"96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073"}
Jul 10 19:01:24.781246 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.781095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" event={"ID":"be1b1971-a498-4212-8b72-12ccafffac4c","Type":"ContainerStarted","Data":"f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666"}
Jul 10 19:01:24.781246 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.781123    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" event={"ID":"be1b1971-a498-4212-8b72-12ccafffac4c","Type":"ContainerStarted","Data":"65f5b8eb4210b5e6cea0c8c5238ab6d4ef42e3265c10fdba41c60221ab81b68a"}
Jul 10 19:01:24.781246 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.781231    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:24.781377 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.781283    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:24.781377 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.781311    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:24.782168 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.782146    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:01:24.782249 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.782146    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:01:24.799943 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:24.799883    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podStartSLOduration=2.799868516 podStartE2EDuration="2.799868516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:24.799372669 +0000 UTC m=+1266.097714738" watchObservedRunningTime="2026-07-10 19:01:24.799868516 +0000 UTC m=+1266.098210585"
Jul 10 19:01:25.616670 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.616634    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:01:25.737337 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.737306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjv4n\" (UniqueName: \"kubernetes.io/projected/382b626a-a7cd-4cd7-a9ba-df3a8638f152-kube-api-access-rjv4n\") pod \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") "
Jul 10 19:01:25.737499 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.737363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/382b626a-a7cd-4cd7-a9ba-df3a8638f152-error-404-isvc-48fe0-kube-rbac-proxy-sar-config\") pod \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") "
Jul 10 19:01:25.737499 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.737445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/382b626a-a7cd-4cd7-a9ba-df3a8638f152-proxy-tls\") pod \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\" (UID: \"382b626a-a7cd-4cd7-a9ba-df3a8638f152\") "
Jul 10 19:01:25.737859 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.737811    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382b626a-a7cd-4cd7-a9ba-df3a8638f152-error-404-isvc-48fe0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-48fe0-kube-rbac-proxy-sar-config") pod "382b626a-a7cd-4cd7-a9ba-df3a8638f152" (UID: "382b626a-a7cd-4cd7-a9ba-df3a8638f152"). InnerVolumeSpecName "error-404-isvc-48fe0-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:25.740491 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.740466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382b626a-a7cd-4cd7-a9ba-df3a8638f152-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "382b626a-a7cd-4cd7-a9ba-df3a8638f152" (UID: "382b626a-a7cd-4cd7-a9ba-df3a8638f152"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:25.740491 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.740480    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382b626a-a7cd-4cd7-a9ba-df3a8638f152-kube-api-access-rjv4n" (OuterVolumeSpecName: "kube-api-access-rjv4n") pod "382b626a-a7cd-4cd7-a9ba-df3a8638f152" (UID: "382b626a-a7cd-4cd7-a9ba-df3a8638f152"). InnerVolumeSpecName "kube-api-access-rjv4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:01:25.788415 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.788379    2565 generic.go:356] "Generic (PLEG): container finished" podID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerID="19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a" exitCode=0
Jul 10 19:01:25.788573 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.788425    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"
Jul 10 19:01:25.788573 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.788459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" event={"ID":"382b626a-a7cd-4cd7-a9ba-df3a8638f152","Type":"ContainerDied","Data":"19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a"}
Jul 10 19:01:25.788573 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.788548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn" event={"ID":"382b626a-a7cd-4cd7-a9ba-df3a8638f152","Type":"ContainerDied","Data":"d9161ac5611e22af94b5f441a9162e592d4b4caba2e31ab185eec787e81ac271"}
Jul 10 19:01:25.788761 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.788585    2565 scope.go:117] "RemoveContainer" containerID="2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4"
Jul 10 19:01:25.790923 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.790900    2565 generic.go:356] "Generic (PLEG): container finished" podID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerID="c5f8864a40c61bcba23be5532aafa5bec188e363012aca67baa2db0082cae758" exitCode=0
Jul 10 19:01:25.791037 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.790992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" event={"ID":"6508d68c-ad79-4a6c-a298-b62e4493cc33","Type":"ContainerDied","Data":"c5f8864a40c61bcba23be5532aafa5bec188e363012aca67baa2db0082cae758"}
Jul 10 19:01:25.791546 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.791517    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:01:25.791643 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.791518    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:01:25.799319 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.799281    2565 scope.go:117] "RemoveContainer" containerID="19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a"
Jul 10 19:01:25.799385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.799370    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:01:25.806797 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.806779    2565 scope.go:117] "RemoveContainer" containerID="2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4"
Jul 10 19:01:25.807106 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:25.807086    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4\": container with ID starting with 2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4 not found: ID does not exist" containerID="2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4"
Jul 10 19:01:25.807161 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.807116    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4"} err="failed to get container status \"2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4\": rpc error: code = NotFound desc = could not find container \"2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4\": container with ID starting with 2b477f174a001013dba4f44625994bbe668a46be31d591e34cb28dcf4434f4f4 not found: ID does not exist"
Jul 10 19:01:25.807161 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.807135    2565 scope.go:117] "RemoveContainer" containerID="19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a"
Jul 10 19:01:25.807386 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:25.807366    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a\": container with ID starting with 19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a not found: ID does not exist" containerID="19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a"
Jul 10 19:01:25.807432 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.807392    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a"} err="failed to get container status \"19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a\": rpc error: code = NotFound desc = could not find container \"19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a\": container with ID starting with 19b9ca10b2cc027c665d8be7011fa4933d88cdbca37c38c7630e2b20a5b8a41a not found: ID does not exist"
Jul 10 19:01:25.813347 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.813316    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"]
Jul 10 19:01:25.818673 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.818653    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48fe0-predictor-87d87457b-zm2hn"]
Jul 10 19:01:25.838364 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.838323    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjv4n\" (UniqueName: \"kubernetes.io/projected/382b626a-a7cd-4cd7-a9ba-df3a8638f152-kube-api-access-rjv4n\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:25.838364 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.838357    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/382b626a-a7cd-4cd7-a9ba-df3a8638f152-error-404-isvc-48fe0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:25.838589 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.838375    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/382b626a-a7cd-4cd7-a9ba-df3a8638f152-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:25.938748 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.938707    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6508d68c-ad79-4a6c-a298-b62e4493cc33-success-200-isvc-48fe0-kube-rbac-proxy-sar-config\") pod \"6508d68c-ad79-4a6c-a298-b62e4493cc33\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") "
Jul 10 19:01:25.938980 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.938779    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls\") pod \"6508d68c-ad79-4a6c-a298-b62e4493cc33\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") "
Jul 10 19:01:25.938980 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.938802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4xb2\" (UniqueName: \"kubernetes.io/projected/6508d68c-ad79-4a6c-a298-b62e4493cc33-kube-api-access-c4xb2\") pod \"6508d68c-ad79-4a6c-a298-b62e4493cc33\" (UID: \"6508d68c-ad79-4a6c-a298-b62e4493cc33\") "
Jul 10 19:01:25.939194 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.939169    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6508d68c-ad79-4a6c-a298-b62e4493cc33-success-200-isvc-48fe0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-48fe0-kube-rbac-proxy-sar-config") pod "6508d68c-ad79-4a6c-a298-b62e4493cc33" (UID: "6508d68c-ad79-4a6c-a298-b62e4493cc33"). InnerVolumeSpecName "success-200-isvc-48fe0-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:25.941042 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.941011    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6508d68c-ad79-4a6c-a298-b62e4493cc33" (UID: "6508d68c-ad79-4a6c-a298-b62e4493cc33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:25.941126 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:25.941040    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6508d68c-ad79-4a6c-a298-b62e4493cc33-kube-api-access-c4xb2" (OuterVolumeSpecName: "kube-api-access-c4xb2") pod "6508d68c-ad79-4a6c-a298-b62e4493cc33" (UID: "6508d68c-ad79-4a6c-a298-b62e4493cc33"). InnerVolumeSpecName "kube-api-access-c4xb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:01:26.040372 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.040293    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6508d68c-ad79-4a6c-a298-b62e4493cc33-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:26.040372 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.040324    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xb2\" (UniqueName: \"kubernetes.io/projected/6508d68c-ad79-4a6c-a298-b62e4493cc33-kube-api-access-c4xb2\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:26.040372 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.040336    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-48fe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6508d68c-ad79-4a6c-a298-b62e4493cc33-success-200-isvc-48fe0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:26.798443 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.798351    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"
Jul 10 19:01:26.798443 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.798356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9" event={"ID":"6508d68c-ad79-4a6c-a298-b62e4493cc33","Type":"ContainerDied","Data":"3d795cefacce5d3b453160d26c17e5883112c1d8527ba971bbee571c651fc46f"}
Jul 10 19:01:26.798897 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.798516    2565 scope.go:117] "RemoveContainer" containerID="6e2c4a70cbef4d6dd8ef800ac061c95fe2b379582e427038cea9bed0484c53e2"
Jul 10 19:01:26.807049 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.807032    2565 scope.go:117] "RemoveContainer" containerID="c5f8864a40c61bcba23be5532aafa5bec188e363012aca67baa2db0082cae758"
Jul 10 19:01:26.819468 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.819431    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"]
Jul 10 19:01:26.823542 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:26.823520    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48fe0-predictor-7c696bdf47-xmhn9"]
Jul 10 19:01:27.274077 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.274045    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" path="/var/lib/kubelet/pods/382b626a-a7cd-4cd7-a9ba-df3a8638f152/volumes"
Jul 10 19:01:27.274467 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.274455    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" path="/var/lib/kubelet/pods/6508d68c-ad79-4a6c-a298-b62e4493cc33/volumes"
Jul 10 19:01:27.910538 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910503    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"]
Jul 10 19:01:27.910891 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910852    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kube-rbac-proxy"
Jul 10 19:01:27.910891 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910863    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kube-rbac-proxy"
Jul 10 19:01:27.910891 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910880    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kube-rbac-proxy"
Jul 10 19:01:27.910891 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910886    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kube-rbac-proxy"
Jul 10 19:01:27.910891 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910892    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container"
Jul 10 19:01:27.911081 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910897    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container"
Jul 10 19:01:27.911081 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container"
Jul 10 19:01:27.911081 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.910916    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container"
Jul 10 19:01:27.911081 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.911002    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kube-rbac-proxy"
Jul 10 19:01:27.911081 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.911014    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kserve-container"
Jul 10 19:01:27.911081 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.911023    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6508d68c-ad79-4a6c-a298-b62e4493cc33" containerName="kube-rbac-proxy"
Jul 10 19:01:27.911081 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.911030    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="382b626a-a7cd-4cd7-a9ba-df3a8638f152" containerName="kserve-container"
Jul 10 19:01:27.915636 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.915614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:27.917981 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.917945    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-9304d-serving-cert\""
Jul 10 19:01:27.917981 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.917967    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-9304d-kube-rbac-proxy-sar-config\""
Jul 10 19:01:27.923193 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.923172    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"]
Jul 10 19:01:27.924854 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:27.924834    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:28.057796 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.057737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743a3274-81fd-4496-8cea-ec2c2a1a1358-proxy-tls\") pod \"sequence-graph-9304d-6745589549-52c2s\" (UID: \"743a3274-81fd-4496-8cea-ec2c2a1a1358\") " pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:28.057974 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.057823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/743a3274-81fd-4496-8cea-ec2c2a1a1358-openshift-service-ca-bundle\") pod \"sequence-graph-9304d-6745589549-52c2s\" (UID: \"743a3274-81fd-4496-8cea-ec2c2a1a1358\") " pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:28.075789 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.075757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:28.159114 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.159074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/743a3274-81fd-4496-8cea-ec2c2a1a1358-openshift-service-ca-bundle\") pod \"sequence-graph-9304d-6745589549-52c2s\" (UID: \"743a3274-81fd-4496-8cea-ec2c2a1a1358\") " pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:28.159297 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.159155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743a3274-81fd-4496-8cea-ec2c2a1a1358-proxy-tls\") pod \"sequence-graph-9304d-6745589549-52c2s\" (UID: \"743a3274-81fd-4496-8cea-ec2c2a1a1358\") " pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:28.159732 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.159708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/743a3274-81fd-4496-8cea-ec2c2a1a1358-openshift-service-ca-bundle\") pod \"sequence-graph-9304d-6745589549-52c2s\" (UID: \"743a3274-81fd-4496-8cea-ec2c2a1a1358\") " pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:28.161637 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.161618    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743a3274-81fd-4496-8cea-ec2c2a1a1358-proxy-tls\") pod \"sequence-graph-9304d-6745589549-52c2s\" (UID: \"743a3274-81fd-4496-8cea-ec2c2a1a1358\") " pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:28.226190 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.226158    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:28.359250 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.359221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"]
Jul 10 19:01:28.361012 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:01:28.360985    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743a3274_81fd_4496_8cea_ec2c2a1a1358.slice/crio-9fd1e91c5b91b6d4b4ccc33ee59536856f93f76f18fed2e73d7b7fc4620bd4c5 WatchSource:0}: Error finding container 9fd1e91c5b91b6d4b4ccc33ee59536856f93f76f18fed2e73d7b7fc4620bd4c5: Status 404 returned error can't find the container with id 9fd1e91c5b91b6d4b4ccc33ee59536856f93f76f18fed2e73d7b7fc4620bd4c5
Jul 10 19:01:28.809925 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.809834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" event={"ID":"743a3274-81fd-4496-8cea-ec2c2a1a1358","Type":"ContainerStarted","Data":"f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024"}
Jul 10 19:01:28.809925 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.809886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" event={"ID":"743a3274-81fd-4496-8cea-ec2c2a1a1358","Type":"ContainerStarted","Data":"9fd1e91c5b91b6d4b4ccc33ee59536856f93f76f18fed2e73d7b7fc4620bd4c5"}
Jul 10 19:01:28.810138 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.809931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:28.829591 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:28.829541    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" podStartSLOduration=1.829520595 podStartE2EDuration="1.829520595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:28.828914133 +0000 UTC m=+1270.127256203" watchObservedRunningTime="2026-07-10 19:01:28.829520595 +0000 UTC m=+1270.127862665"
Jul 10 19:01:29.732118 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:29.732083    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:30.795822 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:30.795780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:01:30.796238 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:30.795877    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:01:30.796360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:30.796330    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:01:30.796561 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:30.796537    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:01:34.732284 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:34.732246    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:34.732661 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:34.732349    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:34.819429 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:34.819399    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:38.127084 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.127043    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"]
Jul 10 19:01:38.127500 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.127239    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d" containerID="cri-o://f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024" gracePeriod=30
Jul 10 19:01:38.173372 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.173340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:38.437389 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.437347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:38.448264 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.448221    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"]
Jul 10 19:01:38.452377 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.452356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.456230 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.456205    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-77aa3-predictor-serving-cert\""
Jul 10 19:01:38.456996 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.456944    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-77aa3-kube-rbac-proxy-sar-config\""
Jul 10 19:01:38.469130 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.469102    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"]
Jul 10 19:01:38.549142 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.549101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-proxy-tls\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.549342 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.549202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-success-200-isvc-77aa3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.549342 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.549260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhmg\" (UniqueName: \"kubernetes.io/projected/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-kube-api-access-zzhmg\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.565232 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.565182    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"]
Jul 10 19:01:38.565454 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.565426    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container" containerID="cri-o://f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a" gracePeriod=30
Jul 10 19:01:38.565518 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.565483    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kube-rbac-proxy" containerID="cri-o://c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77" gracePeriod=30
Jul 10 19:01:38.596080 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.596041    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:38.602583 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.602552    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"]
Jul 10 19:01:38.606306 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.606282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.610389 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.610368    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-77aa3-kube-rbac-proxy-sar-config\""
Jul 10 19:01:38.610773 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.610757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-77aa3-predictor-serving-cert\""
Jul 10 19:01:38.620130 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.620105    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"]
Jul 10 19:01:38.650135 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.650103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-success-200-isvc-77aa3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.650324 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.650152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhmg\" (UniqueName: \"kubernetes.io/projected/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-kube-api-access-zzhmg\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.650324 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.650294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-proxy-tls\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.650986 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.650919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-success-200-isvc-77aa3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.652857 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.652822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-proxy-tls\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.669719 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.669689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhmg\" (UniqueName: \"kubernetes.io/projected/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-kube-api-access-zzhmg\") pod \"success-200-isvc-77aa3-predictor-869969f59f-rj546\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") " pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.700075 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.699992    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:38.756981 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.756400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f20449b-2238-4527-83b3-7917e4a47fcf-error-404-isvc-77aa3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.756981 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.756524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fljd\" (UniqueName: \"kubernetes.io/projected/6f20449b-2238-4527-83b3-7917e4a47fcf-kube-api-access-6fljd\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.756981 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.756568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.764180 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.764117    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:38.804547 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.804513    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"]
Jul 10 19:01:38.804758 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.804732    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container" containerID="cri-o://9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277" gracePeriod=30
Jul 10 19:01:38.804879 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.804846    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kube-rbac-proxy" containerID="cri-o://1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a" gracePeriod=30
Jul 10 19:01:38.852668 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.852630    2565 generic.go:356] "Generic (PLEG): container finished" podID="9bc5a568-590c-4578-b197-840c4a3818fa" containerID="c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77" exitCode=2
Jul 10 19:01:38.852912 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.852712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" event={"ID":"9bc5a568-590c-4578-b197-840c4a3818fa","Type":"ContainerDied","Data":"c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77"}
Jul 10 19:01:38.858770 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.857913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fljd\" (UniqueName: \"kubernetes.io/projected/6f20449b-2238-4527-83b3-7917e4a47fcf-kube-api-access-6fljd\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.858770 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.858008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.858770 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.858068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f20449b-2238-4527-83b3-7917e4a47fcf-error-404-isvc-77aa3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.858770 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:38.858422    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-serving-cert: secret "error-404-isvc-77aa3-predictor-serving-cert" not found
Jul 10 19:01:38.858770 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:38.858484    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls podName:6f20449b-2238-4527-83b3-7917e4a47fcf nodeName:}" failed. No retries permitted until 2026-07-10 19:01:39.358464473 +0000 UTC m=+1280.656806521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls") pod "error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" (UID: "6f20449b-2238-4527-83b3-7917e4a47fcf") : secret "error-404-isvc-77aa3-predictor-serving-cert" not found
Jul 10 19:01:38.859123 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.858971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f20449b-2238-4527-83b3-7917e4a47fcf-error-404-isvc-77aa3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.870697 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.870671    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fljd\" (UniqueName: \"kubernetes.io/projected/6f20449b-2238-4527-83b3-7917e4a47fcf-kube-api-access-6fljd\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:38.873438 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.873404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:38.928353 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:38.928327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"]
Jul 10 19:01:38.929594 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:01:38.929562    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3ef496_3f7c_4ebb_a150_f85279d0f6cc.slice/crio-69f9fe1af72b7bc7b819bad6b89e07fb9b65017656582940c622e0d0eef3ab35 WatchSource:0}: Error finding container 69f9fe1af72b7bc7b819bad6b89e07fb9b65017656582940c622e0d0eef3ab35: Status 404 returned error can't find the container with id 69f9fe1af72b7bc7b819bad6b89e07fb9b65017656582940c622e0d0eef3ab35
Jul 10 19:01:39.044877 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.044849    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:39.362401 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.362316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:39.364865 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.364845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls\") pod \"error-404-isvc-77aa3-predictor-67dd777c88-wqrtn\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") " pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:39.517062 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.517022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:39.656074 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.656027    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"]
Jul 10 19:01:39.657738 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:01:39.657695    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f20449b_2238_4527_83b3_7917e4a47fcf.slice/crio-9d2a20b312f00e7767e9440126ab3bf54eec184758d239bc1b4a3603d3daddf6 WatchSource:0}: Error finding container 9d2a20b312f00e7767e9440126ab3bf54eec184758d239bc1b4a3603d3daddf6: Status 404 returned error can't find the container with id 9d2a20b312f00e7767e9440126ab3bf54eec184758d239bc1b4a3603d3daddf6
Jul 10 19:01:39.732491 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.732452    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:39.817809 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.817771    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:39.861854 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.861811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" event={"ID":"6f20449b-2238-4527-83b3-7917e4a47fcf","Type":"ContainerStarted","Data":"3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46"}
Jul 10 19:01:39.862095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.861895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" event={"ID":"6f20449b-2238-4527-83b3-7917e4a47fcf","Type":"ContainerStarted","Data":"419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529"}
Jul 10 19:01:39.862095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.861935    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:39.862095 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.861997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" event={"ID":"6f20449b-2238-4527-83b3-7917e4a47fcf","Type":"ContainerStarted","Data":"9d2a20b312f00e7767e9440126ab3bf54eec184758d239bc1b4a3603d3daddf6"}
Jul 10 19:01:39.863681 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.863650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" event={"ID":"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc","Type":"ContainerStarted","Data":"553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85"}
Jul 10 19:01:39.863823 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.863708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" event={"ID":"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc","Type":"ContainerStarted","Data":"a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f"}
Jul 10 19:01:39.863823 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.863740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" event={"ID":"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc","Type":"ContainerStarted","Data":"69f9fe1af72b7bc7b819bad6b89e07fb9b65017656582940c622e0d0eef3ab35"}
Jul 10 19:01:39.863823 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.863776    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:39.865602 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.865579    2565 generic.go:356] "Generic (PLEG): container finished" podID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerID="1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a" exitCode=2
Jul 10 19:01:39.865734 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.865636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" event={"ID":"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d","Type":"ContainerDied","Data":"1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a"}
Jul 10 19:01:39.883395 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.883343    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podStartSLOduration=1.883327497 podStartE2EDuration="1.883327497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:39.88293103 +0000 UTC m=+1281.181273101" watchObservedRunningTime="2026-07-10 19:01:39.883327497 +0000 UTC m=+1281.181669566"
Jul 10 19:01:39.901501 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:39.901449    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podStartSLOduration=1.9014175290000002 podStartE2EDuration="1.901417529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:39.901111221 +0000 UTC m=+1281.199453290" watchObservedRunningTime="2026-07-10 19:01:39.901417529 +0000 UTC m=+1281.199759596"
Jul 10 19:01:40.528171 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:40.528133    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 10 19:01:40.797141 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:40.797053    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:01:40.797290 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:40.797053    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:01:40.870570 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:40.870538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:40.870778 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:40.870754    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:40.871825 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:40.871793    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:01:40.871901 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:40.871793    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 10 19:01:41.534737 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:41.534690    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 10 19:01:41.875309 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:41.875209    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 10 19:01:41.875309 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:41.875253    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:01:42.724684 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.724659    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:01:42.881638 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.881605    2565 generic.go:356] "Generic (PLEG): container finished" podID="9bc5a568-590c-4578-b197-840c4a3818fa" containerID="f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a" exitCode=0
Jul 10 19:01:42.881798 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.881652    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"
Jul 10 19:01:42.881798 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.881659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" event={"ID":"9bc5a568-590c-4578-b197-840c4a3818fa","Type":"ContainerDied","Data":"f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a"}
Jul 10 19:01:42.881798 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.881721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5" event={"ID":"9bc5a568-590c-4578-b197-840c4a3818fa","Type":"ContainerDied","Data":"f817e4e1b91eafecebe0e928ab6cf7e22d747973fd2014df930640fecb7f5e92"}
Jul 10 19:01:42.881798 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.881749    2565 scope.go:117] "RemoveContainer" containerID="c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77"
Jul 10 19:01:42.892920 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.892899    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9t64\" (UniqueName: \"kubernetes.io/projected/9bc5a568-590c-4578-b197-840c4a3818fa-kube-api-access-k9t64\") pod \"9bc5a568-590c-4578-b197-840c4a3818fa\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") "
Jul 10 19:01:42.893036 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.892997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bc5a568-590c-4578-b197-840c4a3818fa-proxy-tls\") pod \"9bc5a568-590c-4578-b197-840c4a3818fa\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") "
Jul 10 19:01:42.893087 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.893030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bc5a568-590c-4578-b197-840c4a3818fa-success-200-isvc-9304d-kube-rbac-proxy-sar-config\") pod \"9bc5a568-590c-4578-b197-840c4a3818fa\" (UID: \"9bc5a568-590c-4578-b197-840c4a3818fa\") "
Jul 10 19:01:42.893474 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.893453    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc5a568-590c-4578-b197-840c4a3818fa-success-200-isvc-9304d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-9304d-kube-rbac-proxy-sar-config") pod "9bc5a568-590c-4578-b197-840c4a3818fa" (UID: "9bc5a568-590c-4578-b197-840c4a3818fa"). InnerVolumeSpecName "success-200-isvc-9304d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:42.895075 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.895050    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc5a568-590c-4578-b197-840c4a3818fa-kube-api-access-k9t64" (OuterVolumeSpecName: "kube-api-access-k9t64") pod "9bc5a568-590c-4578-b197-840c4a3818fa" (UID: "9bc5a568-590c-4578-b197-840c4a3818fa"). InnerVolumeSpecName "kube-api-access-k9t64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:01:42.895143 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.895082    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc5a568-590c-4578-b197-840c4a3818fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bc5a568-590c-4578-b197-840c4a3818fa" (UID: "9bc5a568-590c-4578-b197-840c4a3818fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:42.931505 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.931483    2565 scope.go:117] "RemoveContainer" containerID="f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a"
Jul 10 19:01:42.940194 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.940171    2565 scope.go:117] "RemoveContainer" containerID="c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77"
Jul 10 19:01:42.940623 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:42.940599    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77\": container with ID starting with c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77 not found: ID does not exist" containerID="c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77"
Jul 10 19:01:42.940710 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.940639    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77"} err="failed to get container status \"c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77\": rpc error: code = NotFound desc = could not find container \"c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77\": container with ID starting with c2279a062e6ac55b29212a5c1d5ad02b2ca19580d3b252f6ee4b94316cbe5b77 not found: ID does not exist"
Jul 10 19:01:42.940710 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.940665    2565 scope.go:117] "RemoveContainer" containerID="f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a"
Jul 10 19:01:42.941036 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:42.941016    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a\": container with ID starting with f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a not found: ID does not exist" containerID="f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a"
Jul 10 19:01:42.941159 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.941037    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a"} err="failed to get container status \"f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a\": rpc error: code = NotFound desc = could not find container \"f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a\": container with ID starting with f47778ce72f18b90e17db9965e00b4a61509493a9fa3d4bfd0bcaed803200e4a not found: ID does not exist"
Jul 10 19:01:42.994518 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.994488    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9t64\" (UniqueName: \"kubernetes.io/projected/9bc5a568-590c-4578-b197-840c4a3818fa-kube-api-access-k9t64\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:42.994518 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.994517    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bc5a568-590c-4578-b197-840c4a3818fa-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:42.994697 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:42.994533    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bc5a568-590c-4578-b197-840c4a3818fa-success-200-isvc-9304d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:43.059839 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.059809    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:01:43.195479 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.195455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-error-404-isvc-9304d-kube-rbac-proxy-sar-config\") pod \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") "
Jul 10 19:01:43.195559 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.195494    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-proxy-tls\") pod \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") "
Jul 10 19:01:43.195605 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.195578    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpn84\" (UniqueName: \"kubernetes.io/projected/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-kube-api-access-dpn84\") pod \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\" (UID: \"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d\") "
Jul 10 19:01:43.195864 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.195844    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-error-404-isvc-9304d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-9304d-kube-rbac-proxy-sar-config") pod "0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" (UID: "0ee2f1c9-66ce-43fd-9e6f-019f81023f1d"). InnerVolumeSpecName "error-404-isvc-9304d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:43.197649 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.197627    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" (UID: "0ee2f1c9-66ce-43fd-9e6f-019f81023f1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:43.197705 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.197674    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-kube-api-access-dpn84" (OuterVolumeSpecName: "kube-api-access-dpn84") pod "0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" (UID: "0ee2f1c9-66ce-43fd-9e6f-019f81023f1d"). InnerVolumeSpecName "kube-api-access-dpn84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:01:43.212033 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.212004    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"]
Jul 10 19:01:43.219668 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.219642    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9304d-predictor-768c4cd6d5-mrzj5"]
Jul 10 19:01:43.273522 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.273481    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" path="/var/lib/kubelet/pods/9bc5a568-590c-4578-b197-840c4a3818fa/volumes"
Jul 10 19:01:43.296615 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.296586    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpn84\" (UniqueName: \"kubernetes.io/projected/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-kube-api-access-dpn84\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:43.296615 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.296615    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-9304d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-error-404-isvc-9304d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:43.296791 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.296630    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:43.888396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.888305    2565 generic.go:356] "Generic (PLEG): container finished" podID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerID="9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277" exitCode=0
Jul 10 19:01:43.888396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.888371    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"
Jul 10 19:01:43.888902 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.888402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" event={"ID":"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d","Type":"ContainerDied","Data":"9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277"}
Jul 10 19:01:43.888902 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.888457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j" event={"ID":"0ee2f1c9-66ce-43fd-9e6f-019f81023f1d","Type":"ContainerDied","Data":"c1d48e2ffdd5fa875a3f2c3653fc0e427a168bbfc78504bf35a95ed9863fb3a3"}
Jul 10 19:01:43.888902 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.888485    2565 scope.go:117] "RemoveContainer" containerID="1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a"
Jul 10 19:01:43.896873 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.896855    2565 scope.go:117] "RemoveContainer" containerID="9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277"
Jul 10 19:01:43.904427 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.904402    2565 scope.go:117] "RemoveContainer" containerID="1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a"
Jul 10 19:01:43.904688 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:43.904668    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a\": container with ID starting with 1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a not found: ID does not exist" containerID="1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a"
Jul 10 19:01:43.904741 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.904698    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a"} err="failed to get container status \"1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a\": rpc error: code = NotFound desc = could not find container \"1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a\": container with ID starting with 1bdb4995ea62621b0cf174d4b8ec16274735a67802147f750e7436558c5e794a not found: ID does not exist"
Jul 10 19:01:43.904741 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.904716    2565 scope.go:117] "RemoveContainer" containerID="9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277"
Jul 10 19:01:43.904944 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:43.904925    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277\": container with ID starting with 9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277 not found: ID does not exist" containerID="9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277"
Jul 10 19:01:43.905030 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.904976    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277"} err="failed to get container status \"9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277\": rpc error: code = NotFound desc = could not find container \"9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277\": container with ID starting with 9663d80e7a702af5eb7c1b5d90f740c4a75bbc7f8687c2ca65edef23a4a51277 not found: ID does not exist"
Jul 10 19:01:43.914749 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.914722    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"]
Jul 10 19:01:43.919843 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:43.919821    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9304d-predictor-5d68f6d5bb-rsq9j"]
Jul 10 19:01:44.732668 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:44.732624    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:44.817378 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:44.817343    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:45.273796 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:45.273761    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" path="/var/lib/kubelet/pods/0ee2f1c9-66ce-43fd-9e6f-019f81023f1d/volumes"
Jul 10 19:01:46.879975 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:46.879931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:01:46.880421 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:46.880331    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:01:46.880421 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:46.880400    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 10 19:01:46.880796 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:46.880768    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:01:49.732635 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:49.732591    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:49.817678 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:49.817639    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:49.817862 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:49.817756    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:01:50.797057 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:50.797019    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:01:50.797057 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:50.797041    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:01:52.286598 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.286574    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:52.476217 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.476179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601bf731-2dff-4309-ac48-8aeb66d64977-openshift-service-ca-bundle\") pod \"601bf731-2dff-4309-ac48-8aeb66d64977\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") "
Jul 10 19:01:52.476392 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.476247    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls\") pod \"601bf731-2dff-4309-ac48-8aeb66d64977\" (UID: \"601bf731-2dff-4309-ac48-8aeb66d64977\") "
Jul 10 19:01:52.476642 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.476606    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601bf731-2dff-4309-ac48-8aeb66d64977-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "601bf731-2dff-4309-ac48-8aeb66d64977" (UID: "601bf731-2dff-4309-ac48-8aeb66d64977"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:52.478420 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.478398    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "601bf731-2dff-4309-ac48-8aeb66d64977" (UID: "601bf731-2dff-4309-ac48-8aeb66d64977"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:52.576827 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.576792    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601bf731-2dff-4309-ac48-8aeb66d64977-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:52.577008 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.576847    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601bf731-2dff-4309-ac48-8aeb66d64977-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:01:52.933428 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.933393    2565 generic.go:356] "Generic (PLEG): container finished" podID="601bf731-2dff-4309-ac48-8aeb66d64977" containerID="20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db" exitCode=0
Jul 10 19:01:52.933591 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.933440    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"
Jul 10 19:01:52.933591 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.933476    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" event={"ID":"601bf731-2dff-4309-ac48-8aeb66d64977","Type":"ContainerDied","Data":"20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db"}
Jul 10 19:01:52.933591 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.933517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz" event={"ID":"601bf731-2dff-4309-ac48-8aeb66d64977","Type":"ContainerDied","Data":"6f03fdeed89d08c184b0c654725161856281918fd1ccd165930d449e404ed9ad"}
Jul 10 19:01:52.933591 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.933540    2565 scope.go:117] "RemoveContainer" containerID="20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db"
Jul 10 19:01:52.942010 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.941982    2565 scope.go:117] "RemoveContainer" containerID="20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db"
Jul 10 19:01:52.942276 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:01:52.942260    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db\": container with ID starting with 20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db not found: ID does not exist" containerID="20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db"
Jul 10 19:01:52.942328 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.942286    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db"} err="failed to get container status \"20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db\": rpc error: code = NotFound desc = could not find container \"20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db\": container with ID starting with 20278b11d572d985026d612188dde34ea2d8ff6b94dac3845b4b30bc742ff9db not found: ID does not exist"
Jul 10 19:01:52.966257 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.966221    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"]
Jul 10 19:01:52.970503 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:52.970473    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-48fe0-57cb8d9dbf-qqrpz"]
Jul 10 19:01:53.273146 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:53.273108    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" path="/var/lib/kubelet/pods/601bf731-2dff-4309-ac48-8aeb66d64977/volumes"
Jul 10 19:01:54.818046 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:54.818002    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:01:56.880974 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:56.880911    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 10 19:01:56.883201 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:56.880911    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:01:57.080347 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:57.080303    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:01:59.817649 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:01:59.817610    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:02:00.797160 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:00.797124    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:02:00.797340 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:00.797126    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:02:04.817424 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:04.817376    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:02:06.880999 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:06.880932    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 10 19:02:06.881446 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:06.880932    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:02:08.272650 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:08.272624    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:02:08.311787 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:08.311733    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743a3274-81fd-4496-8cea-ec2c2a1a1358-proxy-tls\") pod \"743a3274-81fd-4496-8cea-ec2c2a1a1358\" (UID: \"743a3274-81fd-4496-8cea-ec2c2a1a1358\") "
Jul 10 19:02:08.312330 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:08.311802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/743a3274-81fd-4496-8cea-ec2c2a1a1358-openshift-service-ca-bundle\") pod \"743a3274-81fd-4496-8cea-ec2c2a1a1358\" (UID: \"743a3274-81fd-4496-8cea-ec2c2a1a1358\") "
Jul 10 19:02:08.312439 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:08.312319    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743a3274-81fd-4496-8cea-ec2c2a1a1358-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "743a3274-81fd-4496-8cea-ec2c2a1a1358" (UID: "743a3274-81fd-4496-8cea-ec2c2a1a1358"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:02:08.318076 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:08.318039    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743a3274-81fd-4496-8cea-ec2c2a1a1358-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "743a3274-81fd-4496-8cea-ec2c2a1a1358" (UID: "743a3274-81fd-4496-8cea-ec2c2a1a1358"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:02:08.413386 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:08.413356    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743a3274-81fd-4496-8cea-ec2c2a1a1358-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:02:08.413386 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:08.413388    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/743a3274-81fd-4496-8cea-ec2c2a1a1358-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:02:09.015466 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.015432    2565 generic.go:356] "Generic (PLEG): container finished" podID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerID="f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024" exitCode=0
Jul 10 19:02:09.015636 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.015493    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"
Jul 10 19:02:09.015636 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.015500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" event={"ID":"743a3274-81fd-4496-8cea-ec2c2a1a1358","Type":"ContainerDied","Data":"f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024"}
Jul 10 19:02:09.015636 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.015571    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s" event={"ID":"743a3274-81fd-4496-8cea-ec2c2a1a1358","Type":"ContainerDied","Data":"9fd1e91c5b91b6d4b4ccc33ee59536856f93f76f18fed2e73d7b7fc4620bd4c5"}
Jul 10 19:02:09.015636 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.015593    2565 scope.go:117] "RemoveContainer" containerID="f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024"
Jul 10 19:02:09.024597 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.024571    2565 scope.go:117] "RemoveContainer" containerID="f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024"
Jul 10 19:02:09.024854 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:02:09.024833    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024\": container with ID starting with f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024 not found: ID does not exist" containerID="f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024"
Jul 10 19:02:09.024854 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.024862    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024"} err="failed to get container status \"f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024\": rpc error: code = NotFound desc = could not find container \"f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024\": container with ID starting with f39457c53289d840b392d509836761a4e234ce5852e2cc1ea4253879288d5024 not found: ID does not exist"
Jul 10 19:02:09.041154 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.041129    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"]
Jul 10 19:02:09.045814 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.045791    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9304d-6745589549-52c2s"]
Jul 10 19:02:09.272869 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:09.272783    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" path="/var/lib/kubelet/pods/743a3274-81fd-4496-8cea-ec2c2a1a1358/volumes"
Jul 10 19:02:10.796605 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:10.796565    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:02:10.797063 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:10.796890    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:02:16.880495 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:16.880454    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 10 19:02:16.880893 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:16.880697    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:02:20.797656 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:20.797623    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:02:22.391985 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.391934    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"]
Jul 10 19:02:22.391985 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.391971    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392311    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392323    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392337    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392342    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392350    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kube-rbac-proxy"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392355    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kube-rbac-proxy"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392363    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kube-rbac-proxy"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392369    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kube-rbac-proxy"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392381    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container"
Jul 10 19:02:22.392385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392386    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container"
Jul 10 19:02:22.392669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392394    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d"
Jul 10 19:02:22.392669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392399    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d"
Jul 10 19:02:22.392669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392446    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kserve-container"
Jul 10 19:02:22.392669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392454    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="743a3274-81fd-4496-8cea-ec2c2a1a1358" containerName="sequence-graph-9304d"
Jul 10 19:02:22.392669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392460    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="601bf731-2dff-4309-ac48-8aeb66d64977" containerName="ensemble-graph-48fe0"
Jul 10 19:02:22.392669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392466    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kserve-container"
Jul 10 19:02:22.392669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392472    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc5a568-590c-4578-b197-840c4a3818fa" containerName="kube-rbac-proxy"
Jul 10 19:02:22.392669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.392476    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee2f1c9-66ce-43fd-9e6f-019f81023f1d" containerName="kube-rbac-proxy"
Jul 10 19:02:22.396474 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.396459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:22.399319 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.399300    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-9b322-serving-cert\""
Jul 10 19:02:22.399886 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.399870    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-9b322-kube-rbac-proxy-sar-config\""
Jul 10 19:02:22.407664 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.407641    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"]
Jul 10 19:02:22.538889 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.538853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b636ca-b512-4667-bbd1-fb5a30881024-proxy-tls\") pod \"ensemble-graph-9b322-98666c944-8jgzm\" (UID: \"a1b636ca-b512-4667-bbd1-fb5a30881024\") " pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:22.539110 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.538979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b636ca-b512-4667-bbd1-fb5a30881024-openshift-service-ca-bundle\") pod \"ensemble-graph-9b322-98666c944-8jgzm\" (UID: \"a1b636ca-b512-4667-bbd1-fb5a30881024\") " pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:22.631364 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.631332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:22.640158 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.640134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b636ca-b512-4667-bbd1-fb5a30881024-proxy-tls\") pod \"ensemble-graph-9b322-98666c944-8jgzm\" (UID: \"a1b636ca-b512-4667-bbd1-fb5a30881024\") " pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:22.640283 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.640217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b636ca-b512-4667-bbd1-fb5a30881024-openshift-service-ca-bundle\") pod \"ensemble-graph-9b322-98666c944-8jgzm\" (UID: \"a1b636ca-b512-4667-bbd1-fb5a30881024\") " pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:22.640811 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.640789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b636ca-b512-4667-bbd1-fb5a30881024-openshift-service-ca-bundle\") pod \"ensemble-graph-9b322-98666c944-8jgzm\" (UID: \"a1b636ca-b512-4667-bbd1-fb5a30881024\") " pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:22.642651 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.642611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b636ca-b512-4667-bbd1-fb5a30881024-proxy-tls\") pod \"ensemble-graph-9b322-98666c944-8jgzm\" (UID: \"a1b636ca-b512-4667-bbd1-fb5a30881024\") " pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:22.707586 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.707555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:22.848126 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:22.848096    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"]
Jul 10 19:02:22.850754 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:02:22.850725    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b636ca_b512_4667_bbd1_fb5a30881024.slice/crio-b674d693da6088eaabb029286a70d55e5959ef4f2151f775a33242db7cda689d WatchSource:0}: Error finding container b674d693da6088eaabb029286a70d55e5959ef4f2151f775a33242db7cda689d: Status 404 returned error can't find the container with id b674d693da6088eaabb029286a70d55e5959ef4f2151f775a33242db7cda689d
Jul 10 19:02:23.080437 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:23.080401    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" event={"ID":"a1b636ca-b512-4667-bbd1-fb5a30881024","Type":"ContainerStarted","Data":"1571fbaf36d047a6b5db4dd172249e1f6bc510873b82a327156549e3944cd0b5"}
Jul 10 19:02:23.080625 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:23.080450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" event={"ID":"a1b636ca-b512-4667-bbd1-fb5a30881024","Type":"ContainerStarted","Data":"b674d693da6088eaabb029286a70d55e5959ef4f2151f775a33242db7cda689d"}
Jul 10 19:02:23.080625 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:23.080582    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:23.099053 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:23.099002    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" podStartSLOduration=1.098987373 podStartE2EDuration="1.098987373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:02:23.097740402 +0000 UTC m=+1324.396082504" watchObservedRunningTime="2026-07-10 19:02:23.098987373 +0000 UTC m=+1324.397329432"
Jul 10 19:02:26.880995 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:26.880929    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 10 19:02:26.881635 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:26.881615    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:02:29.091470 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:29.091437    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:02:36.881098 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:36.881071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:02:38.331514 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.331473    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"]
Jul 10 19:02:38.335189 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.335169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:38.338048 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.338030    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-77aa3-kube-rbac-proxy-sar-config\""
Jul 10 19:02:38.338143 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.338034    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-77aa3-serving-cert\""
Jul 10 19:02:38.347601 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.347578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"]
Jul 10 19:02:38.353107 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.353082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:38.453516 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.453480    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:02:38.475679 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.475654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba9870db-7462-44e6-9b16-86b89fcb8e03-proxy-tls\") pod \"sequence-graph-77aa3-6b8c46f78b-8qdqb\" (UID: \"ba9870db-7462-44e6-9b16-86b89fcb8e03\") " pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:38.475832 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.475688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba9870db-7462-44e6-9b16-86b89fcb8e03-openshift-service-ca-bundle\") pod \"sequence-graph-77aa3-6b8c46f78b-8qdqb\" (UID: \"ba9870db-7462-44e6-9b16-86b89fcb8e03\") " pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:38.576578 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.576535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba9870db-7462-44e6-9b16-86b89fcb8e03-proxy-tls\") pod \"sequence-graph-77aa3-6b8c46f78b-8qdqb\" (UID: \"ba9870db-7462-44e6-9b16-86b89fcb8e03\") " pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:38.576578 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.576580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba9870db-7462-44e6-9b16-86b89fcb8e03-openshift-service-ca-bundle\") pod \"sequence-graph-77aa3-6b8c46f78b-8qdqb\" (UID: \"ba9870db-7462-44e6-9b16-86b89fcb8e03\") " pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:38.577244 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.577226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba9870db-7462-44e6-9b16-86b89fcb8e03-openshift-service-ca-bundle\") pod \"sequence-graph-77aa3-6b8c46f78b-8qdqb\" (UID: \"ba9870db-7462-44e6-9b16-86b89fcb8e03\") " pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:38.579233 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.579213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba9870db-7462-44e6-9b16-86b89fcb8e03-proxy-tls\") pod \"sequence-graph-77aa3-6b8c46f78b-8qdqb\" (UID: \"ba9870db-7462-44e6-9b16-86b89fcb8e03\") " pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:38.645376 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.645291    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:38.775989 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:38.775930    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"]
Jul 10 19:02:38.778307 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:02:38.778276    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9870db_7462_44e6_9b16_86b89fcb8e03.slice/crio-b0512179df9432c875097013a6068253b04cfe90cb9d653cbafb4b71abdd61f6 WatchSource:0}: Error finding container b0512179df9432c875097013a6068253b04cfe90cb9d653cbafb4b71abdd61f6: Status 404 returned error can't find the container with id b0512179df9432c875097013a6068253b04cfe90cb9d653cbafb4b71abdd61f6
Jul 10 19:02:39.153537 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:39.153499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" event={"ID":"ba9870db-7462-44e6-9b16-86b89fcb8e03","Type":"ContainerStarted","Data":"33321fe5fdb7bddb6474ef5222c8cbf7b792f18c359fed369d18506efaacf8de"}
Jul 10 19:02:39.153711 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:39.153552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" event={"ID":"ba9870db-7462-44e6-9b16-86b89fcb8e03","Type":"ContainerStarted","Data":"b0512179df9432c875097013a6068253b04cfe90cb9d653cbafb4b71abdd61f6"}
Jul 10 19:02:39.153711 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:39.153636    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:02:39.174360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:39.174308    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" podStartSLOduration=1.174290785 podStartE2EDuration="1.174290785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:02:39.172979307 +0000 UTC m=+1340.471321367" watchObservedRunningTime="2026-07-10 19:02:39.174290785 +0000 UTC m=+1340.472632855"
Jul 10 19:02:45.164148 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:02:45.164065    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:05:19.221972 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:05:19.221925    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:05:19.224083 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:05:19.224059    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:09:16.009339 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:09:16.009295    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:09:17.478591 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:09:17.478558    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:19.249004 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:19.248978    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:10:19.251820 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:19.251788    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:10:33.563670 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:33.563635    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:36.960635 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:36.960604    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"]
Jul 10 19:10:36.961034 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:36.960801    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322" containerID="cri-o://1571fbaf36d047a6b5db4dd172249e1f6bc510873b82a327156549e3944cd0b5" gracePeriod=30
Jul 10 19:10:36.981661 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:36.981624    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:37.127048 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.126988    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:37.180273 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.180243    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"]
Jul 10 19:10:37.183802 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.183782    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.186550 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.186520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-896ee-predictor-serving-cert\""
Jul 10 19:10:37.186713 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.186524    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-896ee-kube-rbac-proxy-sar-config\""
Jul 10 19:10:37.197328 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.197302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"]
Jul 10 19:10:37.207484 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.207456    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"]
Jul 10 19:10:37.207772 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.207731    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container" containerID="cri-o://f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666" gracePeriod=30
Jul 10 19:10:37.207848 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.207822    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kube-rbac-proxy" containerID="cri-o://96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073" gracePeriod=30
Jul 10 19:10:37.209549 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.209511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4xl\" (UniqueName: \"kubernetes.io/projected/b02e8556-f49b-4570-8413-7c49ad0c6c97-kube-api-access-xb4xl\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.209666 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.209577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02e8556-f49b-4570-8413-7c49ad0c6c97-success-200-isvc-896ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.209666 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.209635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02e8556-f49b-4570-8413-7c49ad0c6c97-proxy-tls\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.233876 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.233849    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:37.248422 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.248390    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"]
Jul 10 19:10:37.252001 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.251981    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.254287 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.254259    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-896ee-kube-rbac-proxy-sar-config\""
Jul 10 19:10:37.254287 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.254283    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-896ee-predictor-serving-cert\""
Jul 10 19:10:37.263603 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.263580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"]
Jul 10 19:10:37.303972 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.303926    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"]
Jul 10 19:10:37.304175 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.304149    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" containerID="cri-o://d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d" gracePeriod=30
Jul 10 19:10:37.304242 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.304187    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kube-rbac-proxy" containerID="cri-o://15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec" gracePeriod=30
Jul 10 19:10:37.307699 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.307672    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:37.310254 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.310231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3167c6-5255-44e5-94fc-6f21b754ea80-proxy-tls\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.310410 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.310390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4xl\" (UniqueName: \"kubernetes.io/projected/b02e8556-f49b-4570-8413-7c49ad0c6c97-kube-api-access-xb4xl\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.310486 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.310468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3167c6-5255-44e5-94fc-6f21b754ea80-error-404-isvc-896ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.310544 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.310506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02e8556-f49b-4570-8413-7c49ad0c6c97-success-200-isvc-896ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.310598 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.310550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2kl9\" (UniqueName: \"kubernetes.io/projected/ad3167c6-5255-44e5-94fc-6f21b754ea80-kube-api-access-c2kl9\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.310651 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.310640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02e8556-f49b-4570-8413-7c49ad0c6c97-proxy-tls\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.311485 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.311460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02e8556-f49b-4570-8413-7c49ad0c6c97-success-200-isvc-896ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.313302 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.313275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02e8556-f49b-4570-8413-7c49ad0c6c97-proxy-tls\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.318912 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.318894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4xl\" (UniqueName: \"kubernetes.io/projected/b02e8556-f49b-4570-8413-7c49ad0c6c97-kube-api-access-xb4xl\") pod \"success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") " pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.373126 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.373078    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:37.411594 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.411567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3167c6-5255-44e5-94fc-6f21b754ea80-error-404-isvc-896ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.411691 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.411623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2kl9\" (UniqueName: \"kubernetes.io/projected/ad3167c6-5255-44e5-94fc-6f21b754ea80-kube-api-access-c2kl9\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.411691 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.411676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3167c6-5255-44e5-94fc-6f21b754ea80-proxy-tls\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.412339 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.412309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3167c6-5255-44e5-94fc-6f21b754ea80-error-404-isvc-896ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.414146 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.414127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3167c6-5255-44e5-94fc-6f21b754ea80-proxy-tls\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.419753 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.419729    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2kl9\" (UniqueName: \"kubernetes.io/projected/ad3167c6-5255-44e5-94fc-6f21b754ea80-kube-api-access-c2kl9\") pod \"error-404-isvc-896ee-predictor-5f4cdb868-wh8tt\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") " pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.429123 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.429100    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:37.496229 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.496146    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:37.562854 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.562811    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:37.636333 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.636306    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"]
Jul 10 19:10:37.638978 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:10:37.638931    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02e8556_f49b_4570_8413_7c49ad0c6c97.slice/crio-3ff4aa086de0a0fda71150b7ab5b4139b1cec4f96d0ac8902194746a67af98bf WatchSource:0}: Error finding container 3ff4aa086de0a0fda71150b7ab5b4139b1cec4f96d0ac8902194746a67af98bf: Status 404 returned error can't find the container with id 3ff4aa086de0a0fda71150b7ab5b4139b1cec4f96d0ac8902194746a67af98bf
Jul 10 19:10:37.706608 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:37.706575    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"]
Jul 10 19:10:37.708882 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:10:37.708859    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3167c6_5255_44e5_94fc_6f21b754ea80.slice/crio-5bd6de6f3b1e565799cd762216c6f931d5c46a87169556e02fae7c144dd14b9d WatchSource:0}: Error finding container 5bd6de6f3b1e565799cd762216c6f931d5c46a87169556e02fae7c144dd14b9d: Status 404 returned error can't find the container with id 5bd6de6f3b1e565799cd762216c6f931d5c46a87169556e02fae7c144dd14b9d
Jul 10 19:10:38.126977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.126854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" event={"ID":"b02e8556-f49b-4570-8413-7c49ad0c6c97","Type":"ContainerStarted","Data":"eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe"}
Jul 10 19:10:38.126977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.126935    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" event={"ID":"b02e8556-f49b-4570-8413-7c49ad0c6c97","Type":"ContainerStarted","Data":"8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d"}
Jul 10 19:10:38.127482 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.126986    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:38.127482 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.127017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:38.127482 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.127042    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" event={"ID":"b02e8556-f49b-4570-8413-7c49ad0c6c97","Type":"ContainerStarted","Data":"3ff4aa086de0a0fda71150b7ab5b4139b1cec4f96d0ac8902194746a67af98bf"}
Jul 10 19:10:38.128703 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.128669    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 10 19:10:38.128868 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.128782    2565 generic.go:356] "Generic (PLEG): container finished" podID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerID="15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec" exitCode=2
Jul 10 19:10:38.128868 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.128807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" event={"ID":"e66ec695-62e3-4da4-92d6-4da6f9b69798","Type":"ContainerDied","Data":"15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec"}
Jul 10 19:10:38.130673 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.130651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" event={"ID":"ad3167c6-5255-44e5-94fc-6f21b754ea80","Type":"ContainerStarted","Data":"48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879"}
Jul 10 19:10:38.130776 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.130698    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:38.130776 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.130724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" event={"ID":"ad3167c6-5255-44e5-94fc-6f21b754ea80","Type":"ContainerStarted","Data":"68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d"}
Jul 10 19:10:38.130776 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.130752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" event={"ID":"ad3167c6-5255-44e5-94fc-6f21b754ea80","Type":"ContainerStarted","Data":"5bd6de6f3b1e565799cd762216c6f931d5c46a87169556e02fae7c144dd14b9d"}
Jul 10 19:10:38.130926 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.130831    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:38.131909 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.131886    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 10 19:10:38.132633 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.132614    2565 generic.go:356] "Generic (PLEG): container finished" podID="be1b1971-a498-4212-8b72-12ccafffac4c" containerID="96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073" exitCode=2
Jul 10 19:10:38.132707 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.132655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" event={"ID":"be1b1971-a498-4212-8b72-12ccafffac4c","Type":"ContainerDied","Data":"96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073"}
Jul 10 19:10:38.150354 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.150300    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podStartSLOduration=1.150286493 podStartE2EDuration="1.150286493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:38.148250679 +0000 UTC m=+1819.446592754" watchObservedRunningTime="2026-07-10 19:10:38.150286493 +0000 UTC m=+1819.448628561"
Jul 10 19:10:38.168362 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:38.168299    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podStartSLOduration=1.168280379 podStartE2EDuration="1.168280379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:38.168198733 +0000 UTC m=+1819.466540801" watchObservedRunningTime="2026-07-10 19:10:38.168280379 +0000 UTC m=+1819.466622448"
Jul 10 19:10:39.088666 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:39.088623    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:39.137925 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:39.137884    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 10 19:10:39.138303 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:39.137884    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 10 19:10:40.664729 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.664700    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:10:40.741214 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.741130    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls\") pod \"be1b1971-a498-4212-8b72-12ccafffac4c\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") "
Jul 10 19:10:40.741214 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.741184    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr5ln\" (UniqueName: \"kubernetes.io/projected/be1b1971-a498-4212-8b72-12ccafffac4c-kube-api-access-zr5ln\") pod \"be1b1971-a498-4212-8b72-12ccafffac4c\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") "
Jul 10 19:10:40.741388 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.741216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be1b1971-a498-4212-8b72-12ccafffac4c-success-200-isvc-9b322-kube-rbac-proxy-sar-config\") pod \"be1b1971-a498-4212-8b72-12ccafffac4c\" (UID: \"be1b1971-a498-4212-8b72-12ccafffac4c\") "
Jul 10 19:10:40.741669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.741637    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1b1971-a498-4212-8b72-12ccafffac4c-success-200-isvc-9b322-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-9b322-kube-rbac-proxy-sar-config") pod "be1b1971-a498-4212-8b72-12ccafffac4c" (UID: "be1b1971-a498-4212-8b72-12ccafffac4c"). InnerVolumeSpecName "success-200-isvc-9b322-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:10:40.743283 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.743255    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be1b1971-a498-4212-8b72-12ccafffac4c" (UID: "be1b1971-a498-4212-8b72-12ccafffac4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:10:40.743384 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.743360    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1b1971-a498-4212-8b72-12ccafffac4c-kube-api-access-zr5ln" (OuterVolumeSpecName: "kube-api-access-zr5ln") pod "be1b1971-a498-4212-8b72-12ccafffac4c" (UID: "be1b1971-a498-4212-8b72-12ccafffac4c"). InnerVolumeSpecName "kube-api-access-zr5ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:10:40.791879 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.791833    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 10 19:10:40.796934 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.796894    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:10:40.842530 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.842493    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be1b1971-a498-4212-8b72-12ccafffac4c-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:40.842530 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.842524    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr5ln\" (UniqueName: \"kubernetes.io/projected/be1b1971-a498-4212-8b72-12ccafffac4c-kube-api-access-zr5ln\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:40.842530 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:40.842535    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be1b1971-a498-4212-8b72-12ccafffac4c-success-200-isvc-9b322-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:41.147749 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.147722    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:10:41.147935 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.147914    2565 generic.go:356] "Generic (PLEG): container finished" podID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerID="d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d" exitCode=0
Jul 10 19:10:41.148049 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.147998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" event={"ID":"e66ec695-62e3-4da4-92d6-4da6f9b69798","Type":"ContainerDied","Data":"d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d"}
Jul 10 19:10:41.148107 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.148069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl" event={"ID":"e66ec695-62e3-4da4-92d6-4da6f9b69798","Type":"ContainerDied","Data":"1bb8a4f4cc9dce27b9257ece2880b5fa98e47737784577d8737e24ba483ac72a"}
Jul 10 19:10:41.148153 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.148110    2565 scope.go:117] "RemoveContainer" containerID="15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec"
Jul 10 19:10:41.149871 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.149845    2565 generic.go:356] "Generic (PLEG): container finished" podID="be1b1971-a498-4212-8b72-12ccafffac4c" containerID="f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666" exitCode=0
Jul 10 19:10:41.150024 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.149908    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"
Jul 10 19:10:41.150024 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.149919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" event={"ID":"be1b1971-a498-4212-8b72-12ccafffac4c","Type":"ContainerDied","Data":"f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666"}
Jul 10 19:10:41.150024 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.149986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq" event={"ID":"be1b1971-a498-4212-8b72-12ccafffac4c","Type":"ContainerDied","Data":"65f5b8eb4210b5e6cea0c8c5238ab6d4ef42e3265c10fdba41c60221ab81b68a"}
Jul 10 19:10:41.156854 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.156837    2565 scope.go:117] "RemoveContainer" containerID="d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d"
Jul 10 19:10:41.167077 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.167058    2565 scope.go:117] "RemoveContainer" containerID="15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec"
Jul 10 19:10:41.167422 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:41.167400    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec\": container with ID starting with 15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec not found: ID does not exist" containerID="15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec"
Jul 10 19:10:41.167489 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.167431    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec"} err="failed to get container status \"15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec\": rpc error: code = NotFound desc = could not find container \"15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec\": container with ID starting with 15bb2a07414ad1cde83e596de2ecc851ef09689b6605ddbeab9a4d0b9b1efaec not found: ID does not exist"
Jul 10 19:10:41.167489 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.167451    2565 scope.go:117] "RemoveContainer" containerID="d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d"
Jul 10 19:10:41.167870 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:41.167837    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d\": container with ID starting with d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d not found: ID does not exist" containerID="d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d"
Jul 10 19:10:41.167987 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.167878    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d"} err="failed to get container status \"d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d\": rpc error: code = NotFound desc = could not find container \"d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d\": container with ID starting with d165875cf93db091af94f4ee1be3e4ae02bd29bef916fe3d02d9e6f413d5bc4d not found: ID does not exist"
Jul 10 19:10:41.167987 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.167898    2565 scope.go:117] "RemoveContainer" containerID="96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073"
Jul 10 19:10:41.178008 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.177989    2565 scope.go:117] "RemoveContainer" containerID="f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666"
Jul 10 19:10:41.182834 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.182805    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"]
Jul 10 19:10:41.186814 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.186790    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9b322-predictor-56ff68b897-sfxwq"]
Jul 10 19:10:41.188085 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.188065    2565 scope.go:117] "RemoveContainer" containerID="96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073"
Jul 10 19:10:41.188384 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:41.188367    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073\": container with ID starting with 96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073 not found: ID does not exist" containerID="96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073"
Jul 10 19:10:41.188437 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.188391    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073"} err="failed to get container status \"96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073\": rpc error: code = NotFound desc = could not find container \"96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073\": container with ID starting with 96e8d0f20a02ae897078904e078a8b518a10db72c229d36d10d6aae2750da073 not found: ID does not exist"
Jul 10 19:10:41.188437 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.188410    2565 scope.go:117] "RemoveContainer" containerID="f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666"
Jul 10 19:10:41.188672 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:41.188655    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666\": container with ID starting with f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666 not found: ID does not exist" containerID="f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666"
Jul 10 19:10:41.188726 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.188682    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666"} err="failed to get container status \"f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666\": rpc error: code = NotFound desc = could not find container \"f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666\": container with ID starting with f5024d50c73d3f3af6dd3014a76879e7463dd573dbf776e72445d1b3cb74d666 not found: ID does not exist"
Jul 10 19:10:41.245764 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.245737    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls\") pod \"e66ec695-62e3-4da4-92d6-4da6f9b69798\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") "
Jul 10 19:10:41.245918 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.245789    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nvll\" (UniqueName: \"kubernetes.io/projected/e66ec695-62e3-4da4-92d6-4da6f9b69798-kube-api-access-5nvll\") pod \"e66ec695-62e3-4da4-92d6-4da6f9b69798\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") "
Jul 10 19:10:41.245918 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.245838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66ec695-62e3-4da4-92d6-4da6f9b69798-error-404-isvc-9b322-kube-rbac-proxy-sar-config\") pod \"e66ec695-62e3-4da4-92d6-4da6f9b69798\" (UID: \"e66ec695-62e3-4da4-92d6-4da6f9b69798\") "
Jul 10 19:10:41.246274 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.246251    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66ec695-62e3-4da4-92d6-4da6f9b69798-error-404-isvc-9b322-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-9b322-kube-rbac-proxy-sar-config") pod "e66ec695-62e3-4da4-92d6-4da6f9b69798" (UID: "e66ec695-62e3-4da4-92d6-4da6f9b69798"). InnerVolumeSpecName "error-404-isvc-9b322-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:10:41.247993 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.247973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e66ec695-62e3-4da4-92d6-4da6f9b69798" (UID: "e66ec695-62e3-4da4-92d6-4da6f9b69798"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:10:41.248326 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.248306    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66ec695-62e3-4da4-92d6-4da6f9b69798-kube-api-access-5nvll" (OuterVolumeSpecName: "kube-api-access-5nvll") pod "e66ec695-62e3-4da4-92d6-4da6f9b69798" (UID: "e66ec695-62e3-4da4-92d6-4da6f9b69798"). InnerVolumeSpecName "kube-api-access-5nvll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:10:41.273790 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.273719    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" path="/var/lib/kubelet/pods/be1b1971-a498-4212-8b72-12ccafffac4c/volumes"
Jul 10 19:10:41.347018 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.346972    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-9b322-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66ec695-62e3-4da4-92d6-4da6f9b69798-error-404-isvc-9b322-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:41.347018 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.347011    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66ec695-62e3-4da4-92d6-4da6f9b69798-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:41.347018 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:41.347023    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvll\" (UniqueName: \"kubernetes.io/projected/e66ec695-62e3-4da4-92d6-4da6f9b69798-kube-api-access-5nvll\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:42.156405 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:42.156370    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"
Jul 10 19:10:42.173812 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:42.173780    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"]
Jul 10 19:10:42.175632 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:42.175609    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b322-predictor-657656b8f8-qjlvl"]
Jul 10 19:10:43.274058 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:43.274026    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" path="/var/lib/kubelet/pods/e66ec695-62e3-4da4-92d6-4da6f9b69798/volumes"
Jul 10 19:10:44.087974 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:44.087901    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:44.142232 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:44.142204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:10:44.142399 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:44.142282    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:10:44.142753 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:44.142718    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 10 19:10:44.143054 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:44.143031    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 10 19:10:49.087872 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:49.087832    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:49.088239 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:49.087942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:10:52.975459 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:52.975424    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"]
Jul 10 19:10:52.975940 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:52.975642    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3" containerID="cri-o://33321fe5fdb7bddb6474ef5222c8cbf7b792f18c359fed369d18506efaacf8de" gracePeriod=30
Jul 10 19:10:52.997017 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:52.996984    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:53.084145 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.084110    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"]
Jul 10 19:10:53.084355 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.084330    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" containerID="cri-o://419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529" gracePeriod=30
Jul 10 19:10:53.084407 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.084369    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kube-rbac-proxy" containerID="cri-o://3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46" gracePeriod=30
Jul 10 19:10:53.113743 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.113714    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158306    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"]
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158861    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kube-rbac-proxy"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158882    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kube-rbac-proxy"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158894    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158902    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158924    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158933    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158976    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kube-rbac-proxy"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.158984    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kube-rbac-proxy"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.159089    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kube-rbac-proxy"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.159102    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kserve-container"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.159113    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be1b1971-a498-4212-8b72-12ccafffac4c" containerName="kserve-container"
Jul 10 19:10:53.159747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.159124    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66ec695-62e3-4da4-92d6-4da6f9b69798" containerName="kube-rbac-proxy"
Jul 10 19:10:53.162575 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.162555    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"]
Jul 10 19:10:53.162680 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.162662    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.162772 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.162750    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container" containerID="cri-o://a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f" gracePeriod=30
Jul 10 19:10:53.162811 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.162785    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kube-rbac-proxy" containerID="cri-o://553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85" gracePeriod=30
Jul 10 19:10:53.166612 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.166584    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-df329-predictor-serving-cert\""
Jul 10 19:10:53.166879 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.166859    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-df329-kube-rbac-proxy-sar-config\""
Jul 10 19:10:53.171407 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.171120    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"]
Jul 10 19:10:53.209123 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.209089    2565 generic.go:356] "Generic (PLEG): container finished" podID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerID="3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46" exitCode=2
Jul 10 19:10:53.209296 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.209164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" event={"ID":"6f20449b-2238-4527-83b3-7917e4a47fcf","Type":"ContainerDied","Data":"3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46"}
Jul 10 19:10:53.210078 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.210051    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:53.254974 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.254826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.254974 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.254903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4648106-80c9-4685-9fd2-32e13aad5682-success-200-isvc-df329-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.255243 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.255031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzr45\" (UniqueName: \"kubernetes.io/projected/d4648106-80c9-4685-9fd2-32e13aad5682-kube-api-access-kzr45\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.265281 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.265251    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"]
Jul 10 19:10:53.269087 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.269061    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.273602 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.273579    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-df329-predictor-serving-cert\""
Jul 10 19:10:53.273746 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.273631    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-df329-kube-rbac-proxy-sar-config\""
Jul 10 19:10:53.279549 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.279525    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"]
Jul 10 19:10:53.298402 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.298376    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:53.356173 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.356128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzr45\" (UniqueName: \"kubernetes.io/projected/d4648106-80c9-4685-9fd2-32e13aad5682-kube-api-access-kzr45\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.356355 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.356205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5416e004-2356-42b0-8b4f-e2ee56245acf-error-404-isvc-df329-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.356355 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.356288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zkq\" (UniqueName: \"kubernetes.io/projected/5416e004-2356-42b0-8b4f-e2ee56245acf-kube-api-access-b7zkq\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.356438 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.356368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.356495 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.356433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4648106-80c9-4685-9fd2-32e13aad5682-success-200-isvc-df329-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.356531 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.356494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.356580 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:53.356548    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-df329-predictor-serving-cert: secret "success-200-isvc-df329-predictor-serving-cert" not found
Jul 10 19:10:53.356628 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:53.356611    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls podName:d4648106-80c9-4685-9fd2-32e13aad5682 nodeName:}" failed. No retries permitted until 2026-07-10 19:10:53.856589317 +0000 UTC m=+1835.154931384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls") pod "success-200-isvc-df329-predictor-84bcc6b5bf-96thc" (UID: "d4648106-80c9-4685-9fd2-32e13aad5682") : secret "success-200-isvc-df329-predictor-serving-cert" not found
Jul 10 19:10:53.357265 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.357242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4648106-80c9-4685-9fd2-32e13aad5682-success-200-isvc-df329-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.375697 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.375663    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzr45\" (UniqueName: \"kubernetes.io/projected/d4648106-80c9-4685-9fd2-32e13aad5682-kube-api-access-kzr45\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.457688 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.457648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.457865 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.457708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5416e004-2356-42b0-8b4f-e2ee56245acf-error-404-isvc-df329-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.457865 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.457738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zkq\" (UniqueName: \"kubernetes.io/projected/5416e004-2356-42b0-8b4f-e2ee56245acf-kube-api-access-b7zkq\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.457865 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:53.457814    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-df329-predictor-serving-cert: secret "error-404-isvc-df329-predictor-serving-cert" not found
Jul 10 19:10:53.458031 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:53.457876    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls podName:5416e004-2356-42b0-8b4f-e2ee56245acf nodeName:}" failed. No retries permitted until 2026-07-10 19:10:53.957859504 +0000 UTC m=+1835.256201550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls") pod "error-404-isvc-df329-predictor-7d679b95bb-4zvtz" (UID: "5416e004-2356-42b0-8b4f-e2ee56245acf") : secret "error-404-isvc-df329-predictor-serving-cert" not found
Jul 10 19:10:53.458432 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.458413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5416e004-2356-42b0-8b4f-e2ee56245acf-error-404-isvc-df329-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.481191 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.481152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zkq\" (UniqueName: \"kubernetes.io/projected/5416e004-2356-42b0-8b4f-e2ee56245acf-kube-api-access-b7zkq\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.492160 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.492122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:53.581975 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.581867    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:10:53.861230 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.861136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.863681 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.863646    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls\") pod \"success-200-isvc-df329-predictor-84bcc6b5bf-96thc\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") " pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:53.962295 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.962256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:53.964755 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:53.964724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls\") pod \"error-404-isvc-df329-predictor-7d679b95bb-4zvtz\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") " pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:54.085284 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.085252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:54.088450 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.088417    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:54.142850 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.142808    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 10 19:10:54.143085 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.143058    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 10 19:10:54.181321 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.181265    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:54.216225 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.216194    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerID="553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85" exitCode=2
Jul 10 19:10:54.216370 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.216260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" event={"ID":"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc","Type":"ContainerDied","Data":"553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85"}
Jul 10 19:10:54.237193 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.237138    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"]
Jul 10 19:10:54.336038 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:54.336010    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"]
Jul 10 19:10:54.341085 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:10:54.341051    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5416e004_2356_42b0_8b4f_e2ee56245acf.slice/crio-3f315c7dc8dfeba6c9d567a7a8c9ecd731f5325e7f2b1f0055b583c6c026bf6a WatchSource:0}: Error finding container 3f315c7dc8dfeba6c9d567a7a8c9ecd731f5325e7f2b1f0055b583c6c026bf6a: Status 404 returned error can't find the container with id 3f315c7dc8dfeba6c9d567a7a8c9ecd731f5325e7f2b1f0055b583c6c026bf6a
Jul 10 19:10:55.161663 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.161624    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:55.223394 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.223350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" event={"ID":"5416e004-2356-42b0-8b4f-e2ee56245acf","Type":"ContainerStarted","Data":"b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f"}
Jul 10 19:10:55.223578 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.223469    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:55.223675 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.223650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:10:55.223998 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.223936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" event={"ID":"5416e004-2356-42b0-8b4f-e2ee56245acf","Type":"ContainerStarted","Data":"950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242"}
Jul 10 19:10:55.224087 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.224019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" event={"ID":"5416e004-2356-42b0-8b4f-e2ee56245acf","Type":"ContainerStarted","Data":"3f315c7dc8dfeba6c9d567a7a8c9ecd731f5325e7f2b1f0055b583c6c026bf6a"}
Jul 10 19:10:55.225372 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.225340    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 10 19:10:55.225483 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.225371    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" event={"ID":"d4648106-80c9-4685-9fd2-32e13aad5682","Type":"ContainerStarted","Data":"92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f"}
Jul 10 19:10:55.225483 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.225418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" event={"ID":"d4648106-80c9-4685-9fd2-32e13aad5682","Type":"ContainerStarted","Data":"dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4"}
Jul 10 19:10:55.225483 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.225445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" event={"ID":"d4648106-80c9-4685-9fd2-32e13aad5682","Type":"ContainerStarted","Data":"e9df43ab94a4dfff8ed643fe9c7c2c50fa5127b549538821b7bc44965503023c"}
Jul 10 19:10:55.225658 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.225569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:55.225658 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.225618    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:10:55.226722 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.226613    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 10 19:10:55.255195 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.255130    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podStartSLOduration=2.25511312 podStartE2EDuration="2.25511312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:55.253105227 +0000 UTC m=+1836.551447297" watchObservedRunningTime="2026-07-10 19:10:55.25511312 +0000 UTC m=+1836.553455190"
Jul 10 19:10:55.277246 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:55.277197    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podStartSLOduration=2.277182132 podStartE2EDuration="2.277182132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:55.274629462 +0000 UTC m=+1836.572971557" watchObservedRunningTime="2026-07-10 19:10:55.277182132 +0000 UTC m=+1836.575524200"
Jul 10 19:10:56.230349 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.230310    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 10 19:10:56.230727 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.230323    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 10 19:10:56.819257 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.819229    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:10:56.876015 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.875972    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 10 19:10:56.881302 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.881275    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 10 19:10:56.889583 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.889555    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-success-200-isvc-77aa3-kube-rbac-proxy-sar-config\") pod \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") "
Jul 10 19:10:56.889670 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.889614    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-proxy-tls\") pod \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") "
Jul 10 19:10:56.889716 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.889671    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzhmg\" (UniqueName: \"kubernetes.io/projected/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-kube-api-access-zzhmg\") pod \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\" (UID: \"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc\") "
Jul 10 19:10:56.890012 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.889988    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-success-200-isvc-77aa3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-77aa3-kube-rbac-proxy-sar-config") pod "3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" (UID: "3f3ef496-3f7c-4ebb-a150-f85279d0f6cc"). InnerVolumeSpecName "success-200-isvc-77aa3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:10:56.891988 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.891967    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" (UID: "3f3ef496-3f7c-4ebb-a150-f85279d0f6cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:10:56.892088 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.891990    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-kube-api-access-zzhmg" (OuterVolumeSpecName: "kube-api-access-zzhmg") pod "3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" (UID: "3f3ef496-3f7c-4ebb-a150-f85279d0f6cc"). InnerVolumeSpecName "kube-api-access-zzhmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:10:56.990934 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.990896    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-success-200-isvc-77aa3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:56.990934 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.990927    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:56.990934 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:56.990939    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhmg\" (UniqueName: \"kubernetes.io/projected/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc-kube-api-access-zzhmg\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:10:57.235715 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.235679    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerID="a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f" exitCode=0
Jul 10 19:10:57.236137 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.235733    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"
Jul 10 19:10:57.236137 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.235763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" event={"ID":"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc","Type":"ContainerDied","Data":"a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f"}
Jul 10 19:10:57.236137 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.235830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546" event={"ID":"3f3ef496-3f7c-4ebb-a150-f85279d0f6cc","Type":"ContainerDied","Data":"69f9fe1af72b7bc7b819bad6b89e07fb9b65017656582940c622e0d0eef3ab35"}
Jul 10 19:10:57.236137 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.235861    2565 scope.go:117] "RemoveContainer" containerID="553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85"
Jul 10 19:10:57.236416 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.236394    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 10 19:10:57.245416 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.245399    2565 scope.go:117] "RemoveContainer" containerID="a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f"
Jul 10 19:10:57.260106 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.260081    2565 scope.go:117] "RemoveContainer" containerID="553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85"
Jul 10 19:10:57.260524 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:57.260479    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85\": container with ID starting with 553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85 not found: ID does not exist" containerID="553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85"
Jul 10 19:10:57.260624 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.260541    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85"} err="failed to get container status \"553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85\": rpc error: code = NotFound desc = could not find container \"553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85\": container with ID starting with 553e5e6611e4cecb699a769a609967d27645cd09a93a1f2ef4d01bf500773c85 not found: ID does not exist"
Jul 10 19:10:57.260624 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.260560    2565 scope.go:117] "RemoveContainer" containerID="a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f"
Jul 10 19:10:57.260934 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:10:57.260895    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f\": container with ID starting with a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f not found: ID does not exist" containerID="a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f"
Jul 10 19:10:57.261078 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.260935    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f"} err="failed to get container status \"a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f\": rpc error: code = NotFound desc = could not find container \"a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f\": container with ID starting with a0ce2a057bc814a2cc0c50f1a2eed4bc7628839ca0ee742a2db51c098bf5614f not found: ID does not exist"
Jul 10 19:10:57.263879 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.263858    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"]
Jul 10 19:10:57.272778 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:57.272755    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77aa3-predictor-869969f59f-rj546"]
Jul 10 19:10:59.088652 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:59.088618    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:59.273602 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:10:59.273571    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" path="/var/lib/kubelet/pods/3f3ef496-3f7c-4ebb-a150-f85279d0f6cc/volumes"
Jul 10 19:11:00.161599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:00.161558    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:00.933812 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:00.933786    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:11:01.029511 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.029416    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls\") pod \"6f20449b-2238-4527-83b3-7917e4a47fcf\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") "
Jul 10 19:11:01.029511 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.029461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fljd\" (UniqueName: \"kubernetes.io/projected/6f20449b-2238-4527-83b3-7917e4a47fcf-kube-api-access-6fljd\") pod \"6f20449b-2238-4527-83b3-7917e4a47fcf\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") "
Jul 10 19:11:01.029511 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.029512    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f20449b-2238-4527-83b3-7917e4a47fcf-error-404-isvc-77aa3-kube-rbac-proxy-sar-config\") pod \"6f20449b-2238-4527-83b3-7917e4a47fcf\" (UID: \"6f20449b-2238-4527-83b3-7917e4a47fcf\") "
Jul 10 19:11:01.029970 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.029931    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f20449b-2238-4527-83b3-7917e4a47fcf-error-404-isvc-77aa3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-77aa3-kube-rbac-proxy-sar-config") pod "6f20449b-2238-4527-83b3-7917e4a47fcf" (UID: "6f20449b-2238-4527-83b3-7917e4a47fcf"). InnerVolumeSpecName "error-404-isvc-77aa3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:01.031695 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.031668    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f20449b-2238-4527-83b3-7917e4a47fcf-kube-api-access-6fljd" (OuterVolumeSpecName: "kube-api-access-6fljd") pod "6f20449b-2238-4527-83b3-7917e4a47fcf" (UID: "6f20449b-2238-4527-83b3-7917e4a47fcf"). InnerVolumeSpecName "kube-api-access-6fljd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:11:01.031778 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.031728    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f20449b-2238-4527-83b3-7917e4a47fcf" (UID: "6f20449b-2238-4527-83b3-7917e4a47fcf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:11:01.130441 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.130404    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-77aa3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f20449b-2238-4527-83b3-7917e4a47fcf-error-404-isvc-77aa3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:11:01.130441 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.130436    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f20449b-2238-4527-83b3-7917e4a47fcf-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:11:01.130441 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.130447    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fljd\" (UniqueName: \"kubernetes.io/projected/6f20449b-2238-4527-83b3-7917e4a47fcf-kube-api-access-6fljd\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:11:01.234843 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.234813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:11:01.235305 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.235282    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 10 19:11:01.260096 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.260064    2565 generic.go:356] "Generic (PLEG): container finished" podID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerID="419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529" exitCode=0
Jul 10 19:11:01.260234 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.260117    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"
Jul 10 19:11:01.260234 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.260149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" event={"ID":"6f20449b-2238-4527-83b3-7917e4a47fcf","Type":"ContainerDied","Data":"419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529"}
Jul 10 19:11:01.260234 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.260214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn" event={"ID":"6f20449b-2238-4527-83b3-7917e4a47fcf","Type":"ContainerDied","Data":"9d2a20b312f00e7767e9440126ab3bf54eec184758d239bc1b4a3603d3daddf6"}
Jul 10 19:11:01.260423 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.260247    2565 scope.go:117] "RemoveContainer" containerID="3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46"
Jul 10 19:11:01.272540 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.272407    2565 scope.go:117] "RemoveContainer" containerID="419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529"
Jul 10 19:11:01.280590 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.280569    2565 scope.go:117] "RemoveContainer" containerID="3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46"
Jul 10 19:11:01.280838 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:11:01.280820    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46\": container with ID starting with 3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46 not found: ID does not exist" containerID="3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46"
Jul 10 19:11:01.280884 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.280847    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46"} err="failed to get container status \"3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46\": rpc error: code = NotFound desc = could not find container \"3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46\": container with ID starting with 3261d54913959f25e63eb35f5049fd3f128e5ff004a1228b6193fbed3f186b46 not found: ID does not exist"
Jul 10 19:11:01.280884 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.280865    2565 scope.go:117] "RemoveContainer" containerID="419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529"
Jul 10 19:11:01.281143 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:11:01.281128    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529\": container with ID starting with 419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529 not found: ID does not exist" containerID="419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529"
Jul 10 19:11:01.281191 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.281147    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529"} err="failed to get container status \"419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529\": rpc error: code = NotFound desc = could not find container \"419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529\": container with ID starting with 419abc119f6307c619f307e1cd2cb82716d43cb6e2ec5c95bf8970bae45b1529 not found: ID does not exist"
Jul 10 19:11:01.285265 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.285241    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"]
Jul 10 19:11:01.290574 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:01.290549    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77aa3-predictor-67dd777c88-wqrtn"]
Jul 10 19:11:02.241271 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:02.241243    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:11:02.241856 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:02.241831    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 10 19:11:03.272975 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:03.272924    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" path="/var/lib/kubelet/pods/6f20449b-2238-4527-83b3-7917e4a47fcf/volumes"
Jul 10 19:11:04.087835 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:04.087787    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:04.142867 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:04.142827    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 10 19:11:04.143057 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:04.142977    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 10 19:11:05.161434 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:05.161399    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:05.162155 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:05.161505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:11:07.289054 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.288946    2565 generic.go:356] "Generic (PLEG): container finished" podID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerID="1571fbaf36d047a6b5db4dd172249e1f6bc510873b82a327156549e3944cd0b5" exitCode=0
Jul 10 19:11:07.289054 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.288992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" event={"ID":"a1b636ca-b512-4667-bbd1-fb5a30881024","Type":"ContainerDied","Data":"1571fbaf36d047a6b5db4dd172249e1f6bc510873b82a327156549e3944cd0b5"}
Jul 10 19:11:07.610551 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.610524    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:11:07.790816 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.790787    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b636ca-b512-4667-bbd1-fb5a30881024-openshift-service-ca-bundle\") pod \"a1b636ca-b512-4667-bbd1-fb5a30881024\" (UID: \"a1b636ca-b512-4667-bbd1-fb5a30881024\") "
Jul 10 19:11:07.791015 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.790844    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b636ca-b512-4667-bbd1-fb5a30881024-proxy-tls\") pod \"a1b636ca-b512-4667-bbd1-fb5a30881024\" (UID: \"a1b636ca-b512-4667-bbd1-fb5a30881024\") "
Jul 10 19:11:07.791202 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.791177    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b636ca-b512-4667-bbd1-fb5a30881024-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a1b636ca-b512-4667-bbd1-fb5a30881024" (UID: "a1b636ca-b512-4667-bbd1-fb5a30881024"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:07.793089 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.793062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b636ca-b512-4667-bbd1-fb5a30881024-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1b636ca-b512-4667-bbd1-fb5a30881024" (UID: "a1b636ca-b512-4667-bbd1-fb5a30881024"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:11:07.891755 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.891672    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1b636ca-b512-4667-bbd1-fb5a30881024-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:11:07.891755 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:07.891704    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b636ca-b512-4667-bbd1-fb5a30881024-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:11:08.294863 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:08.294833    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"
Jul 10 19:11:08.294863 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:08.294843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm" event={"ID":"a1b636ca-b512-4667-bbd1-fb5a30881024","Type":"ContainerDied","Data":"b674d693da6088eaabb029286a70d55e5959ef4f2151f775a33242db7cda689d"}
Jul 10 19:11:08.295381 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:08.294900    2565 scope.go:117] "RemoveContainer" containerID="1571fbaf36d047a6b5db4dd172249e1f6bc510873b82a327156549e3944cd0b5"
Jul 10 19:11:08.316285 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:08.316257    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"]
Jul 10 19:11:08.319847 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:08.319825    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9b322-98666c944-8jgzm"]
Jul 10 19:11:09.273887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:09.273852    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" path="/var/lib/kubelet/pods/a1b636ca-b512-4667-bbd1-fb5a30881024/volumes"
Jul 10 19:11:10.161374 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:10.161338    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:11.236167 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:11.236127    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 10 19:11:12.242104 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:12.242058    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 10 19:11:14.142880 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:14.142831    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 10 19:11:14.143275 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:14.142980    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 10 19:11:15.162189 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:15.162140    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:20.162172 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:20.162126    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:21.236286 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:21.236244    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 10 19:11:22.242399 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:22.242361    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 10 19:11:23.370439 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.370351    2565 generic.go:356] "Generic (PLEG): container finished" podID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerID="33321fe5fdb7bddb6474ef5222c8cbf7b792f18c359fed369d18506efaacf8de" exitCode=0
Jul 10 19:11:23.370784 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.370432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" event={"ID":"ba9870db-7462-44e6-9b16-86b89fcb8e03","Type":"ContainerDied","Data":"33321fe5fdb7bddb6474ef5222c8cbf7b792f18c359fed369d18506efaacf8de"}
Jul 10 19:11:23.621820 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.621759    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:11:23.718846 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.718811    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba9870db-7462-44e6-9b16-86b89fcb8e03-openshift-service-ca-bundle\") pod \"ba9870db-7462-44e6-9b16-86b89fcb8e03\" (UID: \"ba9870db-7462-44e6-9b16-86b89fcb8e03\") "
Jul 10 19:11:23.718846 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.718852    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba9870db-7462-44e6-9b16-86b89fcb8e03-proxy-tls\") pod \"ba9870db-7462-44e6-9b16-86b89fcb8e03\" (UID: \"ba9870db-7462-44e6-9b16-86b89fcb8e03\") "
Jul 10 19:11:23.719305 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.719277    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9870db-7462-44e6-9b16-86b89fcb8e03-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ba9870db-7462-44e6-9b16-86b89fcb8e03" (UID: "ba9870db-7462-44e6-9b16-86b89fcb8e03"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:23.721162 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.721141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba9870db-7462-44e6-9b16-86b89fcb8e03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba9870db-7462-44e6-9b16-86b89fcb8e03" (UID: "ba9870db-7462-44e6-9b16-86b89fcb8e03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:11:23.819785 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.819732    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba9870db-7462-44e6-9b16-86b89fcb8e03-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:11:23.819785 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:23.819781    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba9870db-7462-44e6-9b16-86b89fcb8e03-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:11:24.143688 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:24.143650    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 10 19:11:24.144086 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:24.144070    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:11:24.375423 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:24.375396    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"
Jul 10 19:11:24.375423 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:24.375415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb" event={"ID":"ba9870db-7462-44e6-9b16-86b89fcb8e03","Type":"ContainerDied","Data":"b0512179df9432c875097013a6068253b04cfe90cb9d653cbafb4b71abdd61f6"}
Jul 10 19:11:24.375889 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:24.375474    2565 scope.go:117] "RemoveContainer" containerID="33321fe5fdb7bddb6474ef5222c8cbf7b792f18c359fed369d18506efaacf8de"
Jul 10 19:11:24.395904 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:24.395844    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"]
Jul 10 19:11:24.401261 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:24.401240    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77aa3-6b8c46f78b-8qdqb"]
Jul 10 19:11:25.273056 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:25.273022    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" path="/var/lib/kubelet/pods/ba9870db-7462-44e6-9b16-86b89fcb8e03/volumes"
Jul 10 19:11:31.235452 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:31.235411    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 10 19:11:32.242258 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:32.242217    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 10 19:11:34.143549 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:34.143519    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:11:41.235424 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:41.235336    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 10 19:11:42.242773 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:42.242732    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 10 19:11:47.228798 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.228718    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"]
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229102    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229115    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229123    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kube-rbac-proxy"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229129    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kube-rbac-proxy"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229137    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kube-rbac-proxy"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229144    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kube-rbac-proxy"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229151    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229157    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229163    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229168    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229176    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container"
Jul 10 19:11:47.229186 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229181    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container"
Jul 10 19:11:47.229599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229232    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba9870db-7462-44e6-9b16-86b89fcb8e03" containerName="sequence-graph-77aa3"
Jul 10 19:11:47.229599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229241    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kserve-container"
Jul 10 19:11:47.229599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229249    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kserve-container"
Jul 10 19:11:47.229599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229256    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b636ca-b512-4667-bbd1-fb5a30881024" containerName="ensemble-graph-9b322"
Jul 10 19:11:47.229599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229263    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3ef496-3f7c-4ebb-a150-f85279d0f6cc" containerName="kube-rbac-proxy"
Jul 10 19:11:47.229599 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.229268    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f20449b-2238-4527-83b3-7917e4a47fcf" containerName="kube-rbac-proxy"
Jul 10 19:11:47.233653 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.233632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:47.236142 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.236118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-896ee-serving-cert\""
Jul 10 19:11:47.236274 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.236120    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-896ee-kube-rbac-proxy-sar-config\""
Jul 10 19:11:47.247008 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.246983    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"]
Jul 10 19:11:47.248007 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.247983    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:47.319618 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.319583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634087c7-ec7d-4976-aea5-e4adbb715006-proxy-tls\") pod \"splitter-graph-896ee-7c6fcdd6fb-r7wgr\" (UID: \"634087c7-ec7d-4976-aea5-e4adbb715006\") " pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:47.319789 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.319646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634087c7-ec7d-4976-aea5-e4adbb715006-openshift-service-ca-bundle\") pod \"splitter-graph-896ee-7c6fcdd6fb-r7wgr\" (UID: \"634087c7-ec7d-4976-aea5-e4adbb715006\") " pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:47.410907 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.410858    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:47.420962 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.420929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634087c7-ec7d-4976-aea5-e4adbb715006-proxy-tls\") pod \"splitter-graph-896ee-7c6fcdd6fb-r7wgr\" (UID: \"634087c7-ec7d-4976-aea5-e4adbb715006\") " pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:47.421083 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.420996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634087c7-ec7d-4976-aea5-e4adbb715006-openshift-service-ca-bundle\") pod \"splitter-graph-896ee-7c6fcdd6fb-r7wgr\" (UID: \"634087c7-ec7d-4976-aea5-e4adbb715006\") " pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:47.421788 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.421752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634087c7-ec7d-4976-aea5-e4adbb715006-openshift-service-ca-bundle\") pod \"splitter-graph-896ee-7c6fcdd6fb-r7wgr\" (UID: \"634087c7-ec7d-4976-aea5-e4adbb715006\") " pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:47.423471 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.423445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634087c7-ec7d-4976-aea5-e4adbb715006-proxy-tls\") pod \"splitter-graph-896ee-7c6fcdd6fb-r7wgr\" (UID: \"634087c7-ec7d-4976-aea5-e4adbb715006\") " pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:47.545145 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.545057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:47.676523 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:47.676500    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"]
Jul 10 19:11:47.678869 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:11:47.678838    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634087c7_ec7d_4976_aea5_e4adbb715006.slice/crio-2a38fc5b8636eb61e1247c3aff4e01ca16907e70e74af5906db636f87cf98c02 WatchSource:0}: Error finding container 2a38fc5b8636eb61e1247c3aff4e01ca16907e70e74af5906db636f87cf98c02: Status 404 returned error can't find the container with id 2a38fc5b8636eb61e1247c3aff4e01ca16907e70e74af5906db636f87cf98c02
Jul 10 19:11:48.483239 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:48.483205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" event={"ID":"634087c7-ec7d-4976-aea5-e4adbb715006","Type":"ContainerStarted","Data":"2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9"}
Jul 10 19:11:48.483628 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:48.483255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" event={"ID":"634087c7-ec7d-4976-aea5-e4adbb715006","Type":"ContainerStarted","Data":"2a38fc5b8636eb61e1247c3aff4e01ca16907e70e74af5906db636f87cf98c02"}
Jul 10 19:11:48.483628 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:48.483309    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:48.500075 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:48.500025    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" podStartSLOduration=1.500010136 podStartE2EDuration="1.500010136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:48.499174987 +0000 UTC m=+1889.797517056" watchObservedRunningTime="2026-07-10 19:11:48.500010136 +0000 UTC m=+1889.798352205"
Jul 10 19:11:51.236111 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:51.236082    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:11:52.242847 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:52.242818    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:11:54.115559 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:54.115524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:54.216083 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:54.216046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:54.493628 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:54.493595    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:11:57.300780 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.300746    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"]
Jul 10 19:11:57.301257 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.301029    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee" containerID="cri-o://2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9" gracePeriod=30
Jul 10 19:11:57.330473 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.330435    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:57.410245 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.410211    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"]
Jul 10 19:11:57.410503 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.410477    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container" containerID="cri-o://8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d" gracePeriod=30
Jul 10 19:11:57.410567 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.410504    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kube-rbac-proxy" containerID="cri-o://eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe" gracePeriod=30
Jul 10 19:11:57.423903 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.423878    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:57.429878 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.429852    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"]
Jul 10 19:11:57.433677 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.433657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.435934 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.435917    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d1618-predictor-serving-cert\""
Jul 10 19:11:57.436147 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.436132    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d1618-kube-rbac-proxy-sar-config\""
Jul 10 19:11:57.443727 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.443704    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"]
Jul 10 19:11:57.499588 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.499542    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"]
Jul 10 19:11:57.500040 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.499980    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container" containerID="cri-o://68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d" gracePeriod=30
Jul 10 19:11:57.501388 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.501353    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kube-rbac-proxy" containerID="cri-o://48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879" gracePeriod=30
Jul 10 19:11:57.503522 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.503493    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:57.541968 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.541928    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"]
Jul 10 19:11:57.545373 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.545352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.547669 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.547650    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d1618-predictor-serving-cert\""
Jul 10 19:11:57.547776 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.547694    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d1618-kube-rbac-proxy-sar-config\""
Jul 10 19:11:57.555747 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.555723    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"]
Jul 10 19:11:57.575461 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.575416    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:57.606465 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.606440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-success-200-isvc-d1618-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.606626 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.606515    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kntf9\" (UniqueName: \"kubernetes.io/projected/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-kube-api-access-kntf9\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.606626 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.606547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.707126 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.707095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7xj\" (UniqueName: \"kubernetes.io/projected/0158322a-5040-449f-8461-bec846aaeec6-kube-api-access-zm7xj\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.707310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.707145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kntf9\" (UniqueName: \"kubernetes.io/projected/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-kube-api-access-kntf9\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.707310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.707209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.707310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.707247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0158322a-5040-449f-8461-bec846aaeec6-proxy-tls\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.707310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.707295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"success-200-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-success-200-isvc-d1618-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.707442 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:11:57.707324    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d1618-predictor-serving-cert: secret "success-200-isvc-d1618-predictor-serving-cert" not found
Jul 10 19:11:57.707442 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.707352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0158322a-5040-449f-8461-bec846aaeec6-error-404-isvc-d1618-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.707442 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:11:57.707382    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls podName:08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3 nodeName:}" failed. No retries permitted until 2026-07-10 19:11:58.207360993 +0000 UTC m=+1899.505703058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls") pod "success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" (UID: "08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3") : secret "success-200-isvc-d1618-predictor-serving-cert" not found
Jul 10 19:11:57.707929 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.707910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"success-200-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-success-200-isvc-d1618-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.717143 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.717111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kntf9\" (UniqueName: \"kubernetes.io/projected/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-kube-api-access-kntf9\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:57.808423 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.808341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"error-404-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0158322a-5040-449f-8461-bec846aaeec6-error-404-isvc-d1618-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.808423 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.808394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7xj\" (UniqueName: \"kubernetes.io/projected/0158322a-5040-449f-8461-bec846aaeec6-kube-api-access-zm7xj\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.808637 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.808439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0158322a-5040-449f-8461-bec846aaeec6-proxy-tls\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.809300 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.809267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"error-404-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0158322a-5040-449f-8461-bec846aaeec6-error-404-isvc-d1618-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.811104 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.811086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0158322a-5040-449f-8461-bec846aaeec6-proxy-tls\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.819020 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.818998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7xj\" (UniqueName: \"kubernetes.io/projected/0158322a-5040-449f-8461-bec846aaeec6-kube-api-access-zm7xj\") pod \"error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") " pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.834069 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.834043    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:57.857965 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.857919    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:57.932861 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.932827    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:11:57.992420 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:57.992390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"]
Jul 10 19:11:57.994864 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:11:57.994840    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0158322a_5040_449f_8461_bec846aaeec6.slice/crio-023b10106780683a13fa5232e3817b78397accd3398a29af89de1b2bc33704aa WatchSource:0}: Error finding container 023b10106780683a13fa5232e3817b78397accd3398a29af89de1b2bc33704aa: Status 404 returned error can't find the container with id 023b10106780683a13fa5232e3817b78397accd3398a29af89de1b2bc33704aa
Jul 10 19:11:58.213166 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.213120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:58.215723 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.215693    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls\") pod \"success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:58.344003 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.343848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:58.481933 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.481772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"]
Jul 10 19:11:58.484137 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:11:58.484106    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d5dbf1_6092_4fd2_a8c8_10ffcf536ed3.slice/crio-328557966fa54512818327fcdc2baeb429bf027b8da3d17a6698710558cbfb02 WatchSource:0}: Error finding container 328557966fa54512818327fcdc2baeb429bf027b8da3d17a6698710558cbfb02: Status 404 returned error can't find the container with id 328557966fa54512818327fcdc2baeb429bf027b8da3d17a6698710558cbfb02
Jul 10 19:11:58.527940 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.527912    2565 generic.go:356] "Generic (PLEG): container finished" podID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerID="eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe" exitCode=2
Jul 10 19:11:58.528098 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.527993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" event={"ID":"b02e8556-f49b-4570-8413-7c49ad0c6c97","Type":"ContainerDied","Data":"eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe"}
Jul 10 19:11:58.529864 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.529842    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerID="48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879" exitCode=2
Jul 10 19:11:58.529999 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.529907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" event={"ID":"ad3167c6-5255-44e5-94fc-6f21b754ea80","Type":"ContainerDied","Data":"48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879"}
Jul 10 19:11:58.531249 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.531226    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" event={"ID":"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3","Type":"ContainerStarted","Data":"328557966fa54512818327fcdc2baeb429bf027b8da3d17a6698710558cbfb02"}
Jul 10 19:11:58.533110 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.533090    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" event={"ID":"0158322a-5040-449f-8461-bec846aaeec6","Type":"ContainerStarted","Data":"2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02"}
Jul 10 19:11:58.533199 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.533131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" event={"ID":"0158322a-5040-449f-8461-bec846aaeec6","Type":"ContainerStarted","Data":"3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc"}
Jul 10 19:11:58.533199 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.533160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" event={"ID":"0158322a-5040-449f-8461-bec846aaeec6","Type":"ContainerStarted","Data":"023b10106780683a13fa5232e3817b78397accd3398a29af89de1b2bc33704aa"}
Jul 10 19:11:58.533308 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.533281    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:58.551327 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:58.551273    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podStartSLOduration=1.551255517 podStartE2EDuration="1.551255517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:58.550014719 +0000 UTC m=+1899.848356807" watchObservedRunningTime="2026-07-10 19:11:58.551255517 +0000 UTC m=+1899.849597588"
Jul 10 19:11:59.139027 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.138988    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 10 19:11:59.139195 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.138994    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 10 19:11:59.491240 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.491201    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:59.539310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.539272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" event={"ID":"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3","Type":"ContainerStarted","Data":"cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4"}
Jul 10 19:11:59.539521 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.539482    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:59.539602 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.539530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" event={"ID":"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3","Type":"ContainerStarted","Data":"b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4"}
Jul 10 19:11:59.539602 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.539583    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:11:59.539698 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.539619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:11:59.540694 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.540667    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 10 19:11:59.540807 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.540669    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 10 19:11:59.558523 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:11:59.558478    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podStartSLOduration=2.558461384 podStartE2EDuration="2.558461384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:59.557725324 +0000 UTC m=+1900.856067427" watchObservedRunningTime="2026-07-10 19:11:59.558461384 +0000 UTC m=+1900.856803454"
Jul 10 19:12:00.544156 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.544112    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 10 19:12:00.544156 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.544137    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 10 19:12:00.765899 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.765871    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:12:00.856201 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.856171    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:12:00.937575 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.937530    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb4xl\" (UniqueName: \"kubernetes.io/projected/b02e8556-f49b-4570-8413-7c49ad0c6c97-kube-api-access-xb4xl\") pod \"b02e8556-f49b-4570-8413-7c49ad0c6c97\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") "
Jul 10 19:12:00.937767 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.937614    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02e8556-f49b-4570-8413-7c49ad0c6c97-success-200-isvc-896ee-kube-rbac-proxy-sar-config\") pod \"b02e8556-f49b-4570-8413-7c49ad0c6c97\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") "
Jul 10 19:12:00.937767 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.937723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02e8556-f49b-4570-8413-7c49ad0c6c97-proxy-tls\") pod \"b02e8556-f49b-4570-8413-7c49ad0c6c97\" (UID: \"b02e8556-f49b-4570-8413-7c49ad0c6c97\") "
Jul 10 19:12:00.944172 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.938017    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02e8556-f49b-4570-8413-7c49ad0c6c97-success-200-isvc-896ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-896ee-kube-rbac-proxy-sar-config") pod "b02e8556-f49b-4570-8413-7c49ad0c6c97" (UID: "b02e8556-f49b-4570-8413-7c49ad0c6c97"). InnerVolumeSpecName "success-200-isvc-896ee-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:12:00.944765 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.944728    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02e8556-f49b-4570-8413-7c49ad0c6c97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b02e8556-f49b-4570-8413-7c49ad0c6c97" (UID: "b02e8556-f49b-4570-8413-7c49ad0c6c97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:12:00.944886 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:00.944767    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02e8556-f49b-4570-8413-7c49ad0c6c97-kube-api-access-xb4xl" (OuterVolumeSpecName: "kube-api-access-xb4xl") pod "b02e8556-f49b-4570-8413-7c49ad0c6c97" (UID: "b02e8556-f49b-4570-8413-7c49ad0c6c97"). InnerVolumeSpecName "kube-api-access-xb4xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:12:01.039385 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.039349    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3167c6-5255-44e5-94fc-6f21b754ea80-proxy-tls\") pod \"ad3167c6-5255-44e5-94fc-6f21b754ea80\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") "
Jul 10 19:12:01.039574 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.039432    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2kl9\" (UniqueName: \"kubernetes.io/projected/ad3167c6-5255-44e5-94fc-6f21b754ea80-kube-api-access-c2kl9\") pod \"ad3167c6-5255-44e5-94fc-6f21b754ea80\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") "
Jul 10 19:12:01.039574 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.039477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3167c6-5255-44e5-94fc-6f21b754ea80-error-404-isvc-896ee-kube-rbac-proxy-sar-config\") pod \"ad3167c6-5255-44e5-94fc-6f21b754ea80\" (UID: \"ad3167c6-5255-44e5-94fc-6f21b754ea80\") "
Jul 10 19:12:01.039701 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.039635    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb4xl\" (UniqueName: \"kubernetes.io/projected/b02e8556-f49b-4570-8413-7c49ad0c6c97-kube-api-access-xb4xl\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:12:01.039701 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.039655    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02e8556-f49b-4570-8413-7c49ad0c6c97-success-200-isvc-896ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:12:01.039701 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.039670    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02e8556-f49b-4570-8413-7c49ad0c6c97-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:12:01.039890 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.039869    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3167c6-5255-44e5-94fc-6f21b754ea80-error-404-isvc-896ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-896ee-kube-rbac-proxy-sar-config") pod "ad3167c6-5255-44e5-94fc-6f21b754ea80" (UID: "ad3167c6-5255-44e5-94fc-6f21b754ea80"). InnerVolumeSpecName "error-404-isvc-896ee-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:12:01.041555 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.041523    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3167c6-5255-44e5-94fc-6f21b754ea80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad3167c6-5255-44e5-94fc-6f21b754ea80" (UID: "ad3167c6-5255-44e5-94fc-6f21b754ea80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:12:01.041663 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.041615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3167c6-5255-44e5-94fc-6f21b754ea80-kube-api-access-c2kl9" (OuterVolumeSpecName: "kube-api-access-c2kl9") pod "ad3167c6-5255-44e5-94fc-6f21b754ea80" (UID: "ad3167c6-5255-44e5-94fc-6f21b754ea80"). InnerVolumeSpecName "kube-api-access-c2kl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:12:01.141011 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.140927    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2kl9\" (UniqueName: \"kubernetes.io/projected/ad3167c6-5255-44e5-94fc-6f21b754ea80-kube-api-access-c2kl9\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:12:01.141011 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.140981    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-896ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3167c6-5255-44e5-94fc-6f21b754ea80-error-404-isvc-896ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:12:01.141011 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.140993    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3167c6-5255-44e5-94fc-6f21b754ea80-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:12:01.549944 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.549903    2565 generic.go:356] "Generic (PLEG): container finished" podID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerID="8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d" exitCode=0
Jul 10 19:12:01.550851 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.549984    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"
Jul 10 19:12:01.550851 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.549995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" event={"ID":"b02e8556-f49b-4570-8413-7c49ad0c6c97","Type":"ContainerDied","Data":"8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d"}
Jul 10 19:12:01.550851 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.550059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p" event={"ID":"b02e8556-f49b-4570-8413-7c49ad0c6c97","Type":"ContainerDied","Data":"3ff4aa086de0a0fda71150b7ab5b4139b1cec4f96d0ac8902194746a67af98bf"}
Jul 10 19:12:01.550851 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.550102    2565 scope.go:117] "RemoveContainer" containerID="eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe"
Jul 10 19:12:01.552430 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.552403    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerID="68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d" exitCode=0
Jul 10 19:12:01.552621 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.552573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" event={"ID":"ad3167c6-5255-44e5-94fc-6f21b754ea80","Type":"ContainerDied","Data":"68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d"}
Jul 10 19:12:01.552856 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.552831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt" event={"ID":"ad3167c6-5255-44e5-94fc-6f21b754ea80","Type":"ContainerDied","Data":"5bd6de6f3b1e565799cd762216c6f931d5c46a87169556e02fae7c144dd14b9d"}
Jul 10 19:12:01.552947 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.552659    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"
Jul 10 19:12:01.565745 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.565709    2565 scope.go:117] "RemoveContainer" containerID="8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d"
Jul 10 19:12:01.573374 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.573348    2565 scope.go:117] "RemoveContainer" containerID="eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe"
Jul 10 19:12:01.573652 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:12:01.573633    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe\": container with ID starting with eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe not found: ID does not exist" containerID="eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe"
Jul 10 19:12:01.573710 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.573664    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe"} err="failed to get container status \"eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe\": rpc error: code = NotFound desc = could not find container \"eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe\": container with ID starting with eb4dab2d8b734a9421852b0c1a331a6d0d0f21c9b92c6c69d4f2a1e3b6454bbe not found: ID does not exist"
Jul 10 19:12:01.573710 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.573681    2565 scope.go:117] "RemoveContainer" containerID="8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d"
Jul 10 19:12:01.573978 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:12:01.573903    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d\": container with ID starting with 8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d not found: ID does not exist" containerID="8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d"
Jul 10 19:12:01.573978 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.573926    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d"} err="failed to get container status \"8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d\": rpc error: code = NotFound desc = could not find container \"8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d\": container with ID starting with 8ee838a2efcfced4cc72fbde02ca812e7eac95ecbad8eb6ff9baf8418bd28d8d not found: ID does not exist"
Jul 10 19:12:01.573978 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.573939    2565 scope.go:117] "RemoveContainer" containerID="48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879"
Jul 10 19:12:01.577009 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.576983    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"]
Jul 10 19:12:01.578486 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.578468    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-896ee-predictor-7ddbb75dc5-vz24p"]
Jul 10 19:12:01.582543 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.582518    2565 scope.go:117] "RemoveContainer" containerID="68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d"
Jul 10 19:12:01.590218 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.590195    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"]
Jul 10 19:12:01.590587 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.590570    2565 scope.go:117] "RemoveContainer" containerID="48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879"
Jul 10 19:12:01.590862 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:12:01.590842    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879\": container with ID starting with 48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879 not found: ID does not exist" containerID="48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879"
Jul 10 19:12:01.590928 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.590870    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879"} err="failed to get container status \"48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879\": rpc error: code = NotFound desc = could not find container \"48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879\": container with ID starting with 48e4b073b1c728365bbcf2823145d333f5a4ff8c833290b2672d6bb84101d879 not found: ID does not exist"
Jul 10 19:12:01.590928 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.590890    2565 scope.go:117] "RemoveContainer" containerID="68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d"
Jul 10 19:12:01.591163 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:12:01.591146    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d\": container with ID starting with 68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d not found: ID does not exist" containerID="68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d"
Jul 10 19:12:01.591200 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.591170    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d"} err="failed to get container status \"68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d\": rpc error: code = NotFound desc = could not find container \"68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d\": container with ID starting with 68ee14a3caf553f855edb3d2af999aad4b4e483f7429598fb629d29345b7359d not found: ID does not exist"
Jul 10 19:12:01.596647 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:01.596627    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-896ee-predictor-5f4cdb868-wh8tt"]
Jul 10 19:12:03.196408 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196374    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"]
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196722    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kube-rbac-proxy"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196733    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kube-rbac-proxy"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196743    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196749    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196757    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196762    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196779    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kube-rbac-proxy"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196784    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kube-rbac-proxy"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196841    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kube-rbac-proxy"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196851    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" containerName="kserve-container"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196858    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kserve-container"
Jul 10 19:12:03.196969 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.196865    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" containerName="kube-rbac-proxy"
Jul 10 19:12:03.199817 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.199791    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:03.202253 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.202223    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-df329-kube-rbac-proxy-sar-config\""
Jul 10 19:12:03.202365 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.202349    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-df329-serving-cert\""
Jul 10 19:12:03.210283 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.210258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:12:03.210496 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.210479    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"]
Jul 10 19:12:03.273781 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.273746    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad3167c6-5255-44e5-94fc-6f21b754ea80" path="/var/lib/kubelet/pods/ad3167c6-5255-44e5-94fc-6f21b754ea80/volumes"
Jul 10 19:12:03.274191 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.274177    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02e8556-f49b-4570-8413-7c49ad0c6c97" path="/var/lib/kubelet/pods/b02e8556-f49b-4570-8413-7c49ad0c6c97/volumes"
Jul 10 19:12:03.359812 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.359769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/694a1c92-42c8-4daf-872c-8b67671f3100-proxy-tls\") pod \"switch-graph-df329-65d797bb8c-r45qz\" (UID: \"694a1c92-42c8-4daf-872c-8b67671f3100\") " pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:03.359812 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.359816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694a1c92-42c8-4daf-872c-8b67671f3100-openshift-service-ca-bundle\") pod \"switch-graph-df329-65d797bb8c-r45qz\" (UID: \"694a1c92-42c8-4daf-872c-8b67671f3100\") " pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:03.460740 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.460651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/694a1c92-42c8-4daf-872c-8b67671f3100-proxy-tls\") pod \"switch-graph-df329-65d797bb8c-r45qz\" (UID: \"694a1c92-42c8-4daf-872c-8b67671f3100\") " pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:03.460740 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.460695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694a1c92-42c8-4daf-872c-8b67671f3100-openshift-service-ca-bundle\") pod \"switch-graph-df329-65d797bb8c-r45qz\" (UID: \"694a1c92-42c8-4daf-872c-8b67671f3100\") " pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:03.461313 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.461290    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694a1c92-42c8-4daf-872c-8b67671f3100-openshift-service-ca-bundle\") pod \"switch-graph-df329-65d797bb8c-r45qz\" (UID: \"694a1c92-42c8-4daf-872c-8b67671f3100\") " pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:03.463345 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.463324    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/694a1c92-42c8-4daf-872c-8b67671f3100-proxy-tls\") pod \"switch-graph-df329-65d797bb8c-r45qz\" (UID: \"694a1c92-42c8-4daf-872c-8b67671f3100\") " pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:03.488527 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.488482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:12:03.510541 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.510506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:03.643917 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:03.643886    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"]
Jul 10 19:12:03.646522 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:12:03.646494    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694a1c92_42c8_4daf_872c_8b67671f3100.slice/crio-454a7eac50bcb0119bb61b8464a7715f8ce0a1c3ac485e460bae1d37ec7195b4 WatchSource:0}: Error finding container 454a7eac50bcb0119bb61b8464a7715f8ce0a1c3ac485e460bae1d37ec7195b4: Status 404 returned error can't find the container with id 454a7eac50bcb0119bb61b8464a7715f8ce0a1c3ac485e460bae1d37ec7195b4
Jul 10 19:12:04.492142 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:04.492094    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:04.572212 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:04.572182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" event={"ID":"694a1c92-42c8-4daf-872c-8b67671f3100","Type":"ContainerStarted","Data":"c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e"}
Jul 10 19:12:04.572393 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:04.572232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" event={"ID":"694a1c92-42c8-4daf-872c-8b67671f3100","Type":"ContainerStarted","Data":"454a7eac50bcb0119bb61b8464a7715f8ce0a1c3ac485e460bae1d37ec7195b4"}
Jul 10 19:12:04.572393 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:04.572307    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:04.588399 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:04.588338    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" podStartSLOduration=1.588319815 podStartE2EDuration="1.588319815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:12:04.587309822 +0000 UTC m=+1905.885651892" watchObservedRunningTime="2026-07-10 19:12:04.588319815 +0000 UTC m=+1905.886661884"
Jul 10 19:12:05.550180 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:05.550149    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:12:05.550651 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:05.550300    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:12:05.550821 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:05.550773    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 10 19:12:05.550903 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:05.550815    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 10 19:12:09.491177 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:09.491139    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:09.491590 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:09.491253    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:12:10.581718 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:10.581693    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:12:14.491411 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:14.491368    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:15.551715 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:15.551675    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 10 19:12:15.552111 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:15.551675    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 10 19:12:19.492053 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:19.492015    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:24.491538 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:24.491494    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:12:25.550828 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:25.550781    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 10 19:12:25.550828 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:25.550814    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 10 19:12:27.446184 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.446158    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:12:27.557177 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.557089    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634087c7-ec7d-4976-aea5-e4adbb715006-openshift-service-ca-bundle\") pod \"634087c7-ec7d-4976-aea5-e4adbb715006\" (UID: \"634087c7-ec7d-4976-aea5-e4adbb715006\") "
Jul 10 19:12:27.557177 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.557164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634087c7-ec7d-4976-aea5-e4adbb715006-proxy-tls\") pod \"634087c7-ec7d-4976-aea5-e4adbb715006\" (UID: \"634087c7-ec7d-4976-aea5-e4adbb715006\") "
Jul 10 19:12:27.557497 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.557470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634087c7-ec7d-4976-aea5-e4adbb715006-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "634087c7-ec7d-4976-aea5-e4adbb715006" (UID: "634087c7-ec7d-4976-aea5-e4adbb715006"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:12:27.559466 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.559444    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634087c7-ec7d-4976-aea5-e4adbb715006-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "634087c7-ec7d-4976-aea5-e4adbb715006" (UID: "634087c7-ec7d-4976-aea5-e4adbb715006"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:12:27.657910 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.657875    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634087c7-ec7d-4976-aea5-e4adbb715006-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:12:27.657910 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.657908    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634087c7-ec7d-4976-aea5-e4adbb715006-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:12:27.680393 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.680360    2565 generic.go:356] "Generic (PLEG): container finished" podID="634087c7-ec7d-4976-aea5-e4adbb715006" containerID="2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9" exitCode=0
Jul 10 19:12:27.680544 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.680403    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"
Jul 10 19:12:27.680544 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.680412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" event={"ID":"634087c7-ec7d-4976-aea5-e4adbb715006","Type":"ContainerDied","Data":"2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9"}
Jul 10 19:12:27.680544 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.680454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr" event={"ID":"634087c7-ec7d-4976-aea5-e4adbb715006","Type":"ContainerDied","Data":"2a38fc5b8636eb61e1247c3aff4e01ca16907e70e74af5906db636f87cf98c02"}
Jul 10 19:12:27.680544 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.680478    2565 scope.go:117] "RemoveContainer" containerID="2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9"
Jul 10 19:12:27.688974 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.688933    2565 scope.go:117] "RemoveContainer" containerID="2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9"
Jul 10 19:12:27.689240 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:12:27.689224    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9\": container with ID starting with 2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9 not found: ID does not exist" containerID="2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9"
Jul 10 19:12:27.689284 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.689249    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9"} err="failed to get container status \"2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9\": rpc error: code = NotFound desc = could not find container \"2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9\": container with ID starting with 2c4830c0e02c604a48e8b21d76444c3f9ce8739efce53107f067d31604d6e2d9 not found: ID does not exist"
Jul 10 19:12:27.703100 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.703073    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"]
Jul 10 19:12:27.704496 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:27.704476    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-896ee-7c6fcdd6fb-r7wgr"]
Jul 10 19:12:29.273570 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:29.273540    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" path="/var/lib/kubelet/pods/634087c7-ec7d-4976-aea5-e4adbb715006/volumes"
Jul 10 19:12:35.551652 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:35.551607    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 10 19:12:35.551652 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:35.551632    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 10 19:12:45.551584 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:45.551553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:12:45.552057 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:45.551632    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:12:57.513892 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.513856    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"]
Jul 10 19:12:57.516312 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.514257    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee"
Jul 10 19:12:57.516312 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.514270    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee"
Jul 10 19:12:57.516312 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.514322    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="634087c7-ec7d-4976-aea5-e4adbb715006" containerName="splitter-graph-896ee"
Jul 10 19:12:57.516781 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.516757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:12:57.517224 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.517209    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:57.519643 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.519625    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d1618-kube-rbac-proxy-sar-config\""
Jul 10 19:12:57.519727 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.519645    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d1618-serving-cert\""
Jul 10 19:12:57.525321 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.525298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"]
Jul 10 19:12:57.612837 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.612803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caecf54d-8a87-4bca-8d4c-07cdaa107584-openshift-service-ca-bundle\") pod \"splitter-graph-d1618-57bb6975c5-jrx8r\" (UID: \"caecf54d-8a87-4bca-8d4c-07cdaa107584\") " pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:57.613022 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.612886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caecf54d-8a87-4bca-8d4c-07cdaa107584-proxy-tls\") pod \"splitter-graph-d1618-57bb6975c5-jrx8r\" (UID: \"caecf54d-8a87-4bca-8d4c-07cdaa107584\") " pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:57.645315 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.645285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:12:57.713439 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.713409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caecf54d-8a87-4bca-8d4c-07cdaa107584-proxy-tls\") pod \"splitter-graph-d1618-57bb6975c5-jrx8r\" (UID: \"caecf54d-8a87-4bca-8d4c-07cdaa107584\") " pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:57.713630 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.713497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caecf54d-8a87-4bca-8d4c-07cdaa107584-openshift-service-ca-bundle\") pod \"splitter-graph-d1618-57bb6975c5-jrx8r\" (UID: \"caecf54d-8a87-4bca-8d4c-07cdaa107584\") " pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:57.714219 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.714197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caecf54d-8a87-4bca-8d4c-07cdaa107584-openshift-service-ca-bundle\") pod \"splitter-graph-d1618-57bb6975c5-jrx8r\" (UID: \"caecf54d-8a87-4bca-8d4c-07cdaa107584\") " pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:57.715919 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.715900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caecf54d-8a87-4bca-8d4c-07cdaa107584-proxy-tls\") pod \"splitter-graph-d1618-57bb6975c5-jrx8r\" (UID: \"caecf54d-8a87-4bca-8d4c-07cdaa107584\") " pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:57.828637 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.828552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:57.962594 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:57.962558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"]
Jul 10 19:12:57.965830 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:12:57.965805    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaecf54d_8a87_4bca_8d4c_07cdaa107584.slice/crio-18947ac7c0ad5b1efe204a747fb88460c6512cc827679fcbde770975212f5035 WatchSource:0}: Error finding container 18947ac7c0ad5b1efe204a747fb88460c6512cc827679fcbde770975212f5035: Status 404 returned error can't find the container with id 18947ac7c0ad5b1efe204a747fb88460c6512cc827679fcbde770975212f5035
Jul 10 19:12:58.811212 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:58.811174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" event={"ID":"caecf54d-8a87-4bca-8d4c-07cdaa107584","Type":"ContainerStarted","Data":"f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54"}
Jul 10 19:12:58.811639 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:58.811225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" event={"ID":"caecf54d-8a87-4bca-8d4c-07cdaa107584","Type":"ContainerStarted","Data":"18947ac7c0ad5b1efe204a747fb88460c6512cc827679fcbde770975212f5035"}
Jul 10 19:12:58.811639 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:58.811245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:12:58.827606 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:12:58.827550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" podStartSLOduration=1.8275342970000001 podStartE2EDuration="1.827534297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:12:58.826178573 +0000 UTC m=+1960.124520641" watchObservedRunningTime="2026-07-10 19:12:58.827534297 +0000 UTC m=+1960.125876366"
Jul 10 19:13:04.821011 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:13:04.820945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:15:19.276165 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:15:19.276134    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:15:19.280507 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:15:19.280485    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:15:48.659021 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:15:48.658981    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:18:24.913894 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:18:24.913857    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:20:19.305735 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:20:19.305706    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:20:19.310336 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:20:19.310314    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:21:12.185622 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.185535    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"]
Jul 10 19:21:12.186216 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.185759    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618" containerID="cri-o://f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54" gracePeriod=30
Jul 10 19:21:12.244678 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.244647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:21:12.308609 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.308572    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"]
Jul 10 19:21:12.308839 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.308799    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container" containerID="cri-o://b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4" gracePeriod=30
Jul 10 19:21:12.308997 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.308877    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kube-rbac-proxy" containerID="cri-o://cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4" gracePeriod=30
Jul 10 19:21:12.347387 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.347354    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"]
Jul 10 19:21:12.347611 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.347584    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container" containerID="cri-o://3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc" gracePeriod=30
Jul 10 19:21:12.347692 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.347623    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kube-rbac-proxy" containerID="cri-o://2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02" gracePeriod=30
Jul 10 19:21:12.868027 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.867916    2565 generic.go:356] "Generic (PLEG): container finished" podID="0158322a-5040-449f-8461-bec846aaeec6" containerID="2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02" exitCode=2
Jul 10 19:21:12.868027 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.867991    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" event={"ID":"0158322a-5040-449f-8461-bec846aaeec6","Type":"ContainerDied","Data":"2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02"}
Jul 10 19:21:12.869737 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.869710    2565 generic.go:356] "Generic (PLEG): container finished" podID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerID="cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4" exitCode=2
Jul 10 19:21:12.869845 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:12.869746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" event={"ID":"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3","Type":"ContainerDied","Data":"cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4"}
Jul 10 19:21:13.939748 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:13.939710    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:21:14.819342 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:14.819300    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:21:15.620933 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.620911    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:21:15.623683 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.623666    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:21:15.729109 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.729074    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls\") pod \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") "
Jul 10 19:21:15.729109 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.729111    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kntf9\" (UniqueName: \"kubernetes.io/projected/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-kube-api-access-kntf9\") pod \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") "
Jul 10 19:21:15.729363 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.729147    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-success-200-isvc-d1618-kube-rbac-proxy-sar-config\") pod \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\" (UID: \"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3\") "
Jul 10 19:21:15.729363 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.729166    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm7xj\" (UniqueName: \"kubernetes.io/projected/0158322a-5040-449f-8461-bec846aaeec6-kube-api-access-zm7xj\") pod \"0158322a-5040-449f-8461-bec846aaeec6\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") "
Jul 10 19:21:15.729363 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.729182    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0158322a-5040-449f-8461-bec846aaeec6-error-404-isvc-d1618-kube-rbac-proxy-sar-config\") pod \"0158322a-5040-449f-8461-bec846aaeec6\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") "
Jul 10 19:21:15.729363 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.729265    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0158322a-5040-449f-8461-bec846aaeec6-proxy-tls\") pod \"0158322a-5040-449f-8461-bec846aaeec6\" (UID: \"0158322a-5040-449f-8461-bec846aaeec6\") "
Jul 10 19:21:15.729626 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.729595    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-success-200-isvc-d1618-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-d1618-kube-rbac-proxy-sar-config") pod "08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" (UID: "08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3"). InnerVolumeSpecName "success-200-isvc-d1618-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:15.729694 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.729632    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0158322a-5040-449f-8461-bec846aaeec6-error-404-isvc-d1618-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-d1618-kube-rbac-proxy-sar-config") pod "0158322a-5040-449f-8461-bec846aaeec6" (UID: "0158322a-5040-449f-8461-bec846aaeec6"). InnerVolumeSpecName "error-404-isvc-d1618-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:15.731471 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.731450    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" (UID: "08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:15.731566 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.731474    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0158322a-5040-449f-8461-bec846aaeec6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0158322a-5040-449f-8461-bec846aaeec6" (UID: "0158322a-5040-449f-8461-bec846aaeec6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:15.731869 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.731852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0158322a-5040-449f-8461-bec846aaeec6-kube-api-access-zm7xj" (OuterVolumeSpecName: "kube-api-access-zm7xj") pod "0158322a-5040-449f-8461-bec846aaeec6" (UID: "0158322a-5040-449f-8461-bec846aaeec6"). InnerVolumeSpecName "kube-api-access-zm7xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:15.731869 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.731862    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-kube-api-access-kntf9" (OuterVolumeSpecName: "kube-api-access-kntf9") pod "08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" (UID: "08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3"). InnerVolumeSpecName "kube-api-access-kntf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:15.830540 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.830497    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0158322a-5040-449f-8461-bec846aaeec6-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.830540 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.830532    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.830540 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.830542    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kntf9\" (UniqueName: \"kubernetes.io/projected/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-kube-api-access-kntf9\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.830757 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.830552    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3-success-200-isvc-d1618-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.830757 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.830563    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7xj\" (UniqueName: \"kubernetes.io/projected/0158322a-5040-449f-8461-bec846aaeec6-kube-api-access-zm7xj\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.830757 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.830573    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-d1618-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0158322a-5040-449f-8461-bec846aaeec6-error-404-isvc-d1618-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.884173 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.884080    2565 generic.go:356] "Generic (PLEG): container finished" podID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerID="b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4" exitCode=0
Jul 10 19:21:15.884173 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.884131    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"
Jul 10 19:21:15.884395 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.884167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" event={"ID":"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3","Type":"ContainerDied","Data":"b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4"}
Jul 10 19:21:15.884395 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.884231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" event={"ID":"08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3","Type":"ContainerDied","Data":"328557966fa54512818327fcdc2baeb429bf027b8da3d17a6698710558cbfb02"}
Jul 10 19:21:15.884395 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.884269    2565 scope.go:117] "RemoveContainer" containerID="cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4"
Jul 10 19:21:15.885899 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.885870    2565 generic.go:356] "Generic (PLEG): container finished" podID="0158322a-5040-449f-8461-bec846aaeec6" containerID="3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc" exitCode=0
Jul 10 19:21:15.886050 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.885944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" event={"ID":"0158322a-5040-449f-8461-bec846aaeec6","Type":"ContainerDied","Data":"3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc"}
Jul 10 19:21:15.886050 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.886017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" event={"ID":"0158322a-5040-449f-8461-bec846aaeec6","Type":"ContainerDied","Data":"023b10106780683a13fa5232e3817b78397accd3398a29af89de1b2bc33704aa"}
Jul 10 19:21:15.886143 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.886117    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"
Jul 10 19:21:15.894999 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.894981    2565 scope.go:117] "RemoveContainer" containerID="b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4"
Jul 10 19:21:15.902542 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.902524    2565 scope.go:117] "RemoveContainer" containerID="cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4"
Jul 10 19:21:15.902791 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:21:15.902773    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4\": container with ID starting with cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4 not found: ID does not exist" containerID="cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4"
Jul 10 19:21:15.902840 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.902799    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4"} err="failed to get container status \"cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4\": rpc error: code = NotFound desc = could not find container \"cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4\": container with ID starting with cc73643861d91998e9097c5c5cf2006651fdaf2982d66820fe91f72b6d1fbcd4 not found: ID does not exist"
Jul 10 19:21:15.902840 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.902817    2565 scope.go:117] "RemoveContainer" containerID="b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4"
Jul 10 19:21:15.903096 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:21:15.903079    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4\": container with ID starting with b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4 not found: ID does not exist" containerID="b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4"
Jul 10 19:21:15.903156 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.903099    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4"} err="failed to get container status \"b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4\": rpc error: code = NotFound desc = could not find container \"b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4\": container with ID starting with b465941e27a1b7843661725d871c7935ef5001826a8a27bcdc809c94f135eda4 not found: ID does not exist"
Jul 10 19:21:15.903156 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.903112    2565 scope.go:117] "RemoveContainer" containerID="2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02"
Jul 10 19:21:15.906631 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.906612    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"]
Jul 10 19:21:15.910860 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.910768    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp"]
Jul 10 19:21:15.910916 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.910896    2565 scope.go:117] "RemoveContainer" containerID="3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc"
Jul 10 19:21:15.918293 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.918278    2565 scope.go:117] "RemoveContainer" containerID="2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02"
Jul 10 19:21:15.918561 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:21:15.918543    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02\": container with ID starting with 2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02 not found: ID does not exist" containerID="2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02"
Jul 10 19:21:15.918605 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.918570    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02"} err="failed to get container status \"2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02\": rpc error: code = NotFound desc = could not find container \"2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02\": container with ID starting with 2df56121afc4b1f902f1fd4ddb8e547527bfb5fb8ed6761563dd470e5d952b02 not found: ID does not exist"
Jul 10 19:21:15.918605 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.918587    2565 scope.go:117] "RemoveContainer" containerID="3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc"
Jul 10 19:21:15.918820 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:21:15.918805    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc\": container with ID starting with 3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc not found: ID does not exist" containerID="3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc"
Jul 10 19:21:15.918862 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.918823    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc"} err="failed to get container status \"3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc\": rpc error: code = NotFound desc = could not find container \"3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc\": container with ID starting with 3d50ad426caba90011f8509a3d7a8da84f9a0d9084830c4f14820ce2145b2fdc not found: ID does not exist"
Jul 10 19:21:15.921252 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.921232    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"]
Jul 10 19:21:15.924780 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:15.924760    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8"]
Jul 10 19:21:16.544899 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:16.544854    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": context deadline exceeded"
Jul 10 19:21:16.545121 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:16.544860    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": context deadline exceeded"
Jul 10 19:21:16.552058 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:16.552025    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1618-predictor-6bf69b6c7-q5mvp" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: i/o timeout"
Jul 10 19:21:16.552162 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:16.552101    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1618-predictor-6fb8c89d7d-hw5v8" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: i/o timeout"
Jul 10 19:21:17.274336 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:17.274297    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0158322a-5040-449f-8461-bec846aaeec6" path="/var/lib/kubelet/pods/0158322a-5040-449f-8461-bec846aaeec6/volumes"
Jul 10 19:21:17.274799 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:17.274712    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" path="/var/lib/kubelet/pods/08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3/volumes"
Jul 10 19:21:19.819550 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:19.819456    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:21:24.819911 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:24.819874    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:21:24.820316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:24.820000    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:21:29.819334 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:29.819294    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:21:34.819866 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:34.819824    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:21:39.819356 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:39.819306    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:21:42.368376 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.368351    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:21:42.457362 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.457270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caecf54d-8a87-4bca-8d4c-07cdaa107584-proxy-tls\") pod \"caecf54d-8a87-4bca-8d4c-07cdaa107584\" (UID: \"caecf54d-8a87-4bca-8d4c-07cdaa107584\") "
Jul 10 19:21:42.457362 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.457348    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caecf54d-8a87-4bca-8d4c-07cdaa107584-openshift-service-ca-bundle\") pod \"caecf54d-8a87-4bca-8d4c-07cdaa107584\" (UID: \"caecf54d-8a87-4bca-8d4c-07cdaa107584\") "
Jul 10 19:21:42.457695 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.457668    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caecf54d-8a87-4bca-8d4c-07cdaa107584-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "caecf54d-8a87-4bca-8d4c-07cdaa107584" (UID: "caecf54d-8a87-4bca-8d4c-07cdaa107584"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:42.459581 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.459550    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caecf54d-8a87-4bca-8d4c-07cdaa107584-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "caecf54d-8a87-4bca-8d4c-07cdaa107584" (UID: "caecf54d-8a87-4bca-8d4c-07cdaa107584"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:42.558593 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.558561    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caecf54d-8a87-4bca-8d4c-07cdaa107584-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:21:42.558593 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.558589    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caecf54d-8a87-4bca-8d4c-07cdaa107584-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:21:42.996129 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.996088    2565 generic.go:356] "Generic (PLEG): container finished" podID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerID="f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54" exitCode=0
Jul 10 19:21:42.996316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.996153    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"
Jul 10 19:21:42.996316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.996163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" event={"ID":"caecf54d-8a87-4bca-8d4c-07cdaa107584","Type":"ContainerDied","Data":"f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54"}
Jul 10 19:21:42.996316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.996230    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r" event={"ID":"caecf54d-8a87-4bca-8d4c-07cdaa107584","Type":"ContainerDied","Data":"18947ac7c0ad5b1efe204a747fb88460c6512cc827679fcbde770975212f5035"}
Jul 10 19:21:42.996316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:42.996251    2565 scope.go:117] "RemoveContainer" containerID="f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54"
Jul 10 19:21:43.004801 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:43.004781    2565 scope.go:117] "RemoveContainer" containerID="f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54"
Jul 10 19:21:43.005088 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:21:43.005062    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54\": container with ID starting with f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54 not found: ID does not exist" containerID="f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54"
Jul 10 19:21:43.005208 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:43.005094    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54"} err="failed to get container status \"f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54\": rpc error: code = NotFound desc = could not find container \"f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54\": container with ID starting with f4227dde98fc5b59b8b7199d9bedac4879472fb517619bb40cde9f843244cf54 not found: ID does not exist"
Jul 10 19:21:43.016048 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:43.016024    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"]
Jul 10 19:21:43.019312 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:43.019287    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1618-57bb6975c5-jrx8r"]
Jul 10 19:21:43.272817 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:21:43.272733    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" path="/var/lib/kubelet/pods/caecf54d-8a87-4bca-8d4c-07cdaa107584/volumes"
Jul 10 19:23:40.351853 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:23:40.351760    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:24:06.446849 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:24:06.446814    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:25:19.331368 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:25:19.331298    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:25:19.336193 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:25:19.336174    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:28:22.466295 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.466197    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"]
Jul 10 19:28:22.466835 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.466461    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329" containerID="cri-o://c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e" gracePeriod=30
Jul 10 19:28:22.527281 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.527242    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:22.643610 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.643575    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"]
Jul 10 19:28:22.643863 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.643830    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container" containerID="cri-o://dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4" gracePeriod=30
Jul 10 19:28:22.644020 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.643872    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kube-rbac-proxy" containerID="cri-o://92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f" gracePeriod=30
Jul 10 19:28:22.705138 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.705106    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"]
Jul 10 19:28:22.705375 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.705348    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container" containerID="cri-o://950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242" gracePeriod=30
Jul 10 19:28:22.705463 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:22.705383    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kube-rbac-proxy" containerID="cri-o://b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f" gracePeriod=30
Jul 10 19:28:23.563603 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:23.563572    2565 generic.go:356] "Generic (PLEG): container finished" podID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerID="b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f" exitCode=2
Jul 10 19:28:23.564059 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:23.563656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" event={"ID":"5416e004-2356-42b0-8b4f-e2ee56245acf","Type":"ContainerDied","Data":"b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f"}
Jul 10 19:28:23.565224 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:23.565197    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4648106-80c9-4685-9fd2-32e13aad5682" containerID="92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f" exitCode=2
Jul 10 19:28:23.565342 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:23.565262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" event={"ID":"d4648106-80c9-4685-9fd2-32e13aad5682","Type":"ContainerDied","Data":"92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f"}
Jul 10 19:28:23.603184 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:23.603152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:24.164517 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164484    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5gnjq/must-gather-zqx5d"]
Jul 10 19:28:24.164843 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164830    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618"
Jul 10 19:28:24.164887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164845    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618"
Jul 10 19:28:24.164887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164852    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container"
Jul 10 19:28:24.164887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164858    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container"
Jul 10 19:28:24.164887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164865    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container"
Jul 10 19:28:24.164887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164872    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container"
Jul 10 19:28:24.164887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164885    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kube-rbac-proxy"
Jul 10 19:28:24.165103 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164891    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kube-rbac-proxy"
Jul 10 19:28:24.165103 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164899    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kube-rbac-proxy"
Jul 10 19:28:24.165103 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164904    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kube-rbac-proxy"
Jul 10 19:28:24.165103 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164973    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="caecf54d-8a87-4bca-8d4c-07cdaa107584" containerName="splitter-graph-d1618"
Jul 10 19:28:24.165103 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164984    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kserve-container"
Jul 10 19:28:24.165103 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164991    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0158322a-5040-449f-8461-bec846aaeec6" containerName="kube-rbac-proxy"
Jul 10 19:28:24.165103 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.164999    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kserve-container"
Jul 10 19:28:24.165103 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.165004    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d5dbf1-6092-4fd2-a8c8-10ffcf536ed3" containerName="kube-rbac-proxy"
Jul 10 19:28:24.167940 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.167923    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:24.170773 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.170749    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5gnjq\"/\"kube-root-ca.crt\""
Jul 10 19:28:24.170887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.170788    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5gnjq\"/\"openshift-service-ca.crt\""
Jul 10 19:28:24.170887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.170856    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5gnjq\"/\"default-dockercfg-vvvgl\""
Jul 10 19:28:24.175192 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.175095    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5gnjq/must-gather-zqx5d"]
Jul 10 19:28:24.175801 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.175771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:24.229473 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.229435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8rh\" (UniqueName: \"kubernetes.io/projected/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-kube-api-access-zk8rh\") pod \"must-gather-zqx5d\" (UID: \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\") " pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:24.229650 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.229484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-must-gather-output\") pod \"must-gather-zqx5d\" (UID: \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\") " pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:24.246393 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.246358    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:24.330674 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.330634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8rh\" (UniqueName: \"kubernetes.io/projected/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-kube-api-access-zk8rh\") pod \"must-gather-zqx5d\" (UID: \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\") " pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:24.330882 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.330685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-must-gather-output\") pod \"must-gather-zqx5d\" (UID: \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\") " pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:24.331085 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.331065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-must-gather-output\") pod \"must-gather-zqx5d\" (UID: \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\") " pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:24.338912 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.338883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8rh\" (UniqueName: \"kubernetes.io/projected/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-kube-api-access-zk8rh\") pod \"must-gather-zqx5d\" (UID: \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\") " pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:24.486976 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.486850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:24.619995 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.619943    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5gnjq/must-gather-zqx5d"]
Jul 10 19:28:24.622450 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:28:24.622423    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fe7cbb1_4462_4886_aebb_3c774e8dd08f.slice/crio-0a5161e9b51fa805c78fb25ead4674a201a59707d9f86ce0ec28d564dc86a842 WatchSource:0}: Error finding container 0a5161e9b51fa805c78fb25ead4674a201a59707d9f86ce0ec28d564dc86a842: Status 404 returned error can't find the container with id 0a5161e9b51fa805c78fb25ead4674a201a59707d9f86ce0ec28d564dc86a842
Jul 10 19:28:24.624269 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:24.624248    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:28:25.582542 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:25.582473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gnjq/must-gather-zqx5d" event={"ID":"3fe7cbb1-4462-4886-aebb-3c774e8dd08f","Type":"ContainerStarted","Data":"0a5161e9b51fa805c78fb25ead4674a201a59707d9f86ce0ec28d564dc86a842"}
Jul 10 19:28:25.583396 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:25.583145    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:28:26.144883 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.144851    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:28:26.249463 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.249418    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzr45\" (UniqueName: \"kubernetes.io/projected/d4648106-80c9-4685-9fd2-32e13aad5682-kube-api-access-kzr45\") pod \"d4648106-80c9-4685-9fd2-32e13aad5682\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") "
Jul 10 19:28:26.249634 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.249474    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls\") pod \"d4648106-80c9-4685-9fd2-32e13aad5682\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") "
Jul 10 19:28:26.249634 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.249522    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4648106-80c9-4685-9fd2-32e13aad5682-success-200-isvc-df329-kube-rbac-proxy-sar-config\") pod \"d4648106-80c9-4685-9fd2-32e13aad5682\" (UID: \"d4648106-80c9-4685-9fd2-32e13aad5682\") "
Jul 10 19:28:26.250731 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.250698    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4648106-80c9-4685-9fd2-32e13aad5682-success-200-isvc-df329-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-df329-kube-rbac-proxy-sar-config") pod "d4648106-80c9-4685-9fd2-32e13aad5682" (UID: "d4648106-80c9-4685-9fd2-32e13aad5682"). InnerVolumeSpecName "success-200-isvc-df329-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:28:26.257177 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.256321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4648106-80c9-4685-9fd2-32e13aad5682-kube-api-access-kzr45" (OuterVolumeSpecName: "kube-api-access-kzr45") pod "d4648106-80c9-4685-9fd2-32e13aad5682" (UID: "d4648106-80c9-4685-9fd2-32e13aad5682"). InnerVolumeSpecName "kube-api-access-kzr45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:28:26.259415 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.259380    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4648106-80c9-4685-9fd2-32e13aad5682" (UID: "d4648106-80c9-4685-9fd2-32e13aad5682"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:28:26.292263 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.292090    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:28:26.350240 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.350202    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzr45\" (UniqueName: \"kubernetes.io/projected/d4648106-80c9-4685-9fd2-32e13aad5682-kube-api-access-kzr45\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:26.350240 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.350242    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4648106-80c9-4685-9fd2-32e13aad5682-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:26.350485 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.350263    2565 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4648106-80c9-4685-9fd2-32e13aad5682-success-200-isvc-df329-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:26.451113 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.451073    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5416e004-2356-42b0-8b4f-e2ee56245acf-error-404-isvc-df329-kube-rbac-proxy-sar-config\") pod \"5416e004-2356-42b0-8b4f-e2ee56245acf\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") "
Jul 10 19:28:26.451307 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.451133    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls\") pod \"5416e004-2356-42b0-8b4f-e2ee56245acf\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") "
Jul 10 19:28:26.451307 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.451169    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7zkq\" (UniqueName: \"kubernetes.io/projected/5416e004-2356-42b0-8b4f-e2ee56245acf-kube-api-access-b7zkq\") pod \"5416e004-2356-42b0-8b4f-e2ee56245acf\" (UID: \"5416e004-2356-42b0-8b4f-e2ee56245acf\") "
Jul 10 19:28:26.451694 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.451655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5416e004-2356-42b0-8b4f-e2ee56245acf-error-404-isvc-df329-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-df329-kube-rbac-proxy-sar-config") pod "5416e004-2356-42b0-8b4f-e2ee56245acf" (UID: "5416e004-2356-42b0-8b4f-e2ee56245acf"). InnerVolumeSpecName "error-404-isvc-df329-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:28:26.455487 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.455455    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5416e004-2356-42b0-8b4f-e2ee56245acf-kube-api-access-b7zkq" (OuterVolumeSpecName: "kube-api-access-b7zkq") pod "5416e004-2356-42b0-8b4f-e2ee56245acf" (UID: "5416e004-2356-42b0-8b4f-e2ee56245acf"). InnerVolumeSpecName "kube-api-access-b7zkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:28:26.455820 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.455783    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5416e004-2356-42b0-8b4f-e2ee56245acf" (UID: "5416e004-2356-42b0-8b4f-e2ee56245acf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:28:26.552353 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.552319    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zkq\" (UniqueName: \"kubernetes.io/projected/5416e004-2356-42b0-8b4f-e2ee56245acf-kube-api-access-b7zkq\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:26.552353 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.552354    2565 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-df329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5416e004-2356-42b0-8b4f-e2ee56245acf-error-404-isvc-df329-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:26.552600 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.552372    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5416e004-2356-42b0-8b4f-e2ee56245acf-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:26.592625 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.592318    2565 generic.go:356] "Generic (PLEG): container finished" podID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerID="950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242" exitCode=0
Jul 10 19:28:26.592625 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.592423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" event={"ID":"5416e004-2356-42b0-8b4f-e2ee56245acf","Type":"ContainerDied","Data":"950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242"}
Jul 10 19:28:26.592625 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.592482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz" event={"ID":"5416e004-2356-42b0-8b4f-e2ee56245acf","Type":"ContainerDied","Data":"3f315c7dc8dfeba6c9d567a7a8c9ecd731f5325e7f2b1f0055b583c6c026bf6a"}
Jul 10 19:28:26.592625 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.592521    2565 scope.go:117] "RemoveContainer" containerID="b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f"
Jul 10 19:28:26.592976 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.592662    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"
Jul 10 19:28:26.596605 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.596095    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4648106-80c9-4685-9fd2-32e13aad5682" containerID="dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4" exitCode=0
Jul 10 19:28:26.596605 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.596153    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"
Jul 10 19:28:26.596605 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.596168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" event={"ID":"d4648106-80c9-4685-9fd2-32e13aad5682","Type":"ContainerDied","Data":"dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4"}
Jul 10 19:28:26.596605 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.596219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc" event={"ID":"d4648106-80c9-4685-9fd2-32e13aad5682","Type":"ContainerDied","Data":"e9df43ab94a4dfff8ed643fe9c7c2c50fa5127b549538821b7bc44965503023c"}
Jul 10 19:28:26.606088 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.606067    2565 scope.go:117] "RemoveContainer" containerID="950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242"
Jul 10 19:28:26.617787 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.617768    2565 scope.go:117] "RemoveContainer" containerID="b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f"
Jul 10 19:28:26.618154 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:26.618125    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f\": container with ID starting with b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f not found: ID does not exist" containerID="b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f"
Jul 10 19:28:26.618245 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.618167    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f"} err="failed to get container status \"b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f\": rpc error: code = NotFound desc = could not find container \"b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f\": container with ID starting with b25bc0112f15683c0ece532c1482e6ba088d8d0d0141a7a84f8c149ee276334f not found: ID does not exist"
Jul 10 19:28:26.618245 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.618193    2565 scope.go:117] "RemoveContainer" containerID="950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242"
Jul 10 19:28:26.618499 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:26.618469    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242\": container with ID starting with 950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242 not found: ID does not exist" containerID="950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242"
Jul 10 19:28:26.618575 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.618509    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242"} err="failed to get container status \"950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242\": rpc error: code = NotFound desc = could not find container \"950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242\": container with ID starting with 950486ba587f1b3c72784d7b8e0c354edcfc9b90c0b6212561fcee6f725f4242 not found: ID does not exist"
Jul 10 19:28:26.618575 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.618536    2565 scope.go:117] "RemoveContainer" containerID="92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f"
Jul 10 19:28:26.620150 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.620132    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"]
Jul 10 19:28:26.623188 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.623169    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df329-predictor-7d679b95bb-4zvtz"]
Jul 10 19:28:26.628082 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.628056    2565 scope.go:117] "RemoveContainer" containerID="dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4"
Jul 10 19:28:26.634775 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.634747    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"]
Jul 10 19:28:26.637087 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.637070    2565 scope.go:117] "RemoveContainer" containerID="92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f"
Jul 10 19:28:26.637439 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:26.637415    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f\": container with ID starting with 92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f not found: ID does not exist" containerID="92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f"
Jul 10 19:28:26.637532 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.637448    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f"} err="failed to get container status \"92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f\": rpc error: code = NotFound desc = could not find container \"92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f\": container with ID starting with 92b4278d2636255fd6d0444fecb5d4cacd39912e734ed223f80b61d3184b6f2f not found: ID does not exist"
Jul 10 19:28:26.637532 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.637473    2565 scope.go:117] "RemoveContainer" containerID="dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4"
Jul 10 19:28:26.637828 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:26.637810    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4\": container with ID starting with dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4 not found: ID does not exist" containerID="dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4"
Jul 10 19:28:26.637893 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.637836    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4"} err="failed to get container status \"dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4\": rpc error: code = NotFound desc = could not find container \"dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4\": container with ID starting with dfffc23c125f5a84cd56ae2f68b9dfe971d19f28cdb75979482d0e9f5636b6a4 not found: ID does not exist"
Jul 10 19:28:26.638248 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:26.638228    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df329-predictor-84bcc6b5bf-96thc"]
Jul 10 19:28:27.274178 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:27.274139    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" path="/var/lib/kubelet/pods/5416e004-2356-42b0-8b4f-e2ee56245acf/volumes"
Jul 10 19:28:27.274663 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:27.274645    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" path="/var/lib/kubelet/pods/d4648106-80c9-4685-9fd2-32e13aad5682/volumes"
Jul 10 19:28:30.580900 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:30.580862    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:28:31.621841 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:31.621807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gnjq/must-gather-zqx5d" event={"ID":"3fe7cbb1-4462-4886-aebb-3c774e8dd08f","Type":"ContainerStarted","Data":"2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d"}
Jul 10 19:28:31.622231 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:31.621852    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gnjq/must-gather-zqx5d" event={"ID":"3fe7cbb1-4462-4886-aebb-3c774e8dd08f","Type":"ContainerStarted","Data":"b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542"}
Jul 10 19:28:31.638645 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:31.638595    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5gnjq/must-gather-zqx5d" podStartSLOduration=1.4262105649999999 podStartE2EDuration="7.638559225s" totalImagesPullingTime="6.21234866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:28:31.637544821 +0000 UTC m=+2892.935886894" watchObservedRunningTime="2026-07-10 19:28:31.638559225 +0000 UTC m=+2892.936901293"
Jul 10 19:28:35.581670 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:35.581631    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:28:35.586031 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:35.581737    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:28:39.930755 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:39.930715    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:40.580284 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:40.580239    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:28:40.812975 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:40.812902    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:41.639782 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:41.639749    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:42.434988 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:42.434933    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:43.229154 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:43.229121    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:44.098327 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:44.098287    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:44.888515 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:44.888472    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:45.581885 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:45.581838    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:28:45.693295 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:45.693261    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:46.483410 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:46.483348    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:47.303459 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:47.303409    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:48.125347 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:48.125317    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:48.940782 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:48.940738    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-df329-65d797bb8c-r45qz_694a1c92-42c8-4daf-872c-8b67671f3100/switch-graph-df329/0.log"
Jul 10 19:28:50.580719 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:50.580675    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:28:52.619701 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.619676    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:28:52.670010 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.669974    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/694a1c92-42c8-4daf-872c-8b67671f3100-proxy-tls\") pod \"694a1c92-42c8-4daf-872c-8b67671f3100\" (UID: \"694a1c92-42c8-4daf-872c-8b67671f3100\") "
Jul 10 19:28:52.670181 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.670042    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694a1c92-42c8-4daf-872c-8b67671f3100-openshift-service-ca-bundle\") pod \"694a1c92-42c8-4daf-872c-8b67671f3100\" (UID: \"694a1c92-42c8-4daf-872c-8b67671f3100\") "
Jul 10 19:28:52.670392 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.670371    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694a1c92-42c8-4daf-872c-8b67671f3100-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "694a1c92-42c8-4daf-872c-8b67671f3100" (UID: "694a1c92-42c8-4daf-872c-8b67671f3100"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:28:52.672233 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.672212    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694a1c92-42c8-4daf-872c-8b67671f3100-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "694a1c92-42c8-4daf-872c-8b67671f3100" (UID: "694a1c92-42c8-4daf-872c-8b67671f3100"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:28:52.715727 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.715694    2565 generic.go:356] "Generic (PLEG): container finished" podID="694a1c92-42c8-4daf-872c-8b67671f3100" containerID="c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e" exitCode=0
Jul 10 19:28:52.715922 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.715743    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"
Jul 10 19:28:52.715922 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.715773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" event={"ID":"694a1c92-42c8-4daf-872c-8b67671f3100","Type":"ContainerDied","Data":"c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e"}
Jul 10 19:28:52.715922 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.715832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz" event={"ID":"694a1c92-42c8-4daf-872c-8b67671f3100","Type":"ContainerDied","Data":"454a7eac50bcb0119bb61b8464a7715f8ce0a1c3ac485e460bae1d37ec7195b4"}
Jul 10 19:28:52.715922 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.715863    2565 scope.go:117] "RemoveContainer" containerID="c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e"
Jul 10 19:28:52.725907 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.725807    2565 generic.go:356] "Generic (PLEG): container finished" podID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerID="b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542" exitCode=0
Jul 10 19:28:52.726046 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.725933    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gnjq/must-gather-zqx5d" event={"ID":"3fe7cbb1-4462-4886-aebb-3c774e8dd08f","Type":"ContainerDied","Data":"b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542"}
Jul 10 19:28:52.726367 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.726339    2565 scope.go:117] "RemoveContainer" containerID="b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542"
Jul 10 19:28:52.730102 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.730081    2565 scope.go:117] "RemoveContainer" containerID="c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e"
Jul 10 19:28:52.730394 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:52.730371    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e\": container with ID starting with c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e not found: ID does not exist" containerID="c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e"
Jul 10 19:28:52.730448 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.730408    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e"} err="failed to get container status \"c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e\": rpc error: code = NotFound desc = could not find container \"c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e\": container with ID starting with c4636d89bf670ac311eb689c169631849aa7651a8c941f84bdca63e35830115e not found: ID does not exist"
Jul 10 19:28:52.751377 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.751352    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"]
Jul 10 19:28:52.755312 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.755285    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df329-65d797bb8c-r45qz"]
Jul 10 19:28:52.771355 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.771330    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694a1c92-42c8-4daf-872c-8b67671f3100-openshift-service-ca-bundle\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:52.771355 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:52.771352    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/694a1c92-42c8-4daf-872c-8b67671f3100-proxy-tls\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:53.242787 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.242757    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5gnjq_must-gather-zqx5d_3fe7cbb1-4462-4886-aebb-3c774e8dd08f/gather/0.log"
Jul 10 19:28:53.272864 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.272825    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" path="/var/lib/kubelet/pods/694a1c92-42c8-4daf-872c-8b67671f3100/volumes"
Jul 10 19:28:53.678751 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.678717    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:53.814489 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814453    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9qxct/must-gather-cr5pn"]
Jul 10 19:28:53.814761 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814749    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329"
Jul 10 19:28:53.814801 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814763    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329"
Jul 10 19:28:53.814801 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814772    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kube-rbac-proxy"
Jul 10 19:28:53.814801 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814778    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kube-rbac-proxy"
Jul 10 19:28:53.814801 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814787    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container"
Jul 10 19:28:53.814801 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814793    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814808    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kube-rbac-proxy"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814813    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kube-rbac-proxy"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814821    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814826    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814870    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kube-rbac-proxy"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814877    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kube-rbac-proxy"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814883    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4648106-80c9-4685-9fd2-32e13aad5682" containerName="kserve-container"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814888    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5416e004-2356-42b0-8b4f-e2ee56245acf" containerName="kserve-container"
Jul 10 19:28:53.814977 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.814892    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="694a1c92-42c8-4daf-872c-8b67671f3100" containerName="switch-graph-df329"
Jul 10 19:28:53.817579 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.817559    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qxct/must-gather-cr5pn"
Jul 10 19:28:53.820241 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.820218    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9qxct\"/\"openshift-service-ca.crt\""
Jul 10 19:28:53.820337 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.820218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9qxct\"/\"default-dockercfg-htsxz\""
Jul 10 19:28:53.821262 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.821248    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9qxct\"/\"kube-root-ca.crt\""
Jul 10 19:28:53.827734 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.827711    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9qxct/must-gather-cr5pn"]
Jul 10 19:28:53.834911 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.834886    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:53.877918 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.877878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46n8m\" (UniqueName: \"kubernetes.io/projected/2ee00290-f312-4760-81e7-cc3a818c0c6a-kube-api-access-46n8m\") pod \"must-gather-cr5pn\" (UID: \"2ee00290-f312-4760-81e7-cc3a818c0c6a\") " pod="openshift-must-gather-9qxct/must-gather-cr5pn"
Jul 10 19:28:53.878119 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.877925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ee00290-f312-4760-81e7-cc3a818c0c6a-must-gather-output\") pod \"must-gather-cr5pn\" (UID: \"2ee00290-f312-4760-81e7-cc3a818c0c6a\") " pod="openshift-must-gather-9qxct/must-gather-cr5pn"
Jul 10 19:28:53.908427 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.908385    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:53.978982 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.978863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46n8m\" (UniqueName: \"kubernetes.io/projected/2ee00290-f312-4760-81e7-cc3a818c0c6a-kube-api-access-46n8m\") pod \"must-gather-cr5pn\" (UID: \"2ee00290-f312-4760-81e7-cc3a818c0c6a\") " pod="openshift-must-gather-9qxct/must-gather-cr5pn"
Jul 10 19:28:53.978982 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.978914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ee00290-f312-4760-81e7-cc3a818c0c6a-must-gather-output\") pod \"must-gather-cr5pn\" (UID: \"2ee00290-f312-4760-81e7-cc3a818c0c6a\") " pod="openshift-must-gather-9qxct/must-gather-cr5pn"
Jul 10 19:28:53.979299 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.979281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ee00290-f312-4760-81e7-cc3a818c0c6a-must-gather-output\") pod \"must-gather-cr5pn\" (UID: \"2ee00290-f312-4760-81e7-cc3a818c0c6a\") " pod="openshift-must-gather-9qxct/must-gather-cr5pn"
Jul 10 19:28:53.987075 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:53.987042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46n8m\" (UniqueName: \"kubernetes.io/projected/2ee00290-f312-4760-81e7-cc3a818c0c6a-kube-api-access-46n8m\") pod \"must-gather-cr5pn\" (UID: \"2ee00290-f312-4760-81e7-cc3a818c0c6a\") " pod="openshift-must-gather-9qxct/must-gather-cr5pn"
Jul 10 19:28:54.126278 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:54.126243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qxct/must-gather-cr5pn"
Jul 10 19:28:54.256567 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:54.256541    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9qxct/must-gather-cr5pn"]
Jul 10 19:28:54.259054 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:28:54.259029    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ee00290_f312_4760_81e7_cc3a818c0c6a.slice/crio-9bc2907ad607884790bc84cc1e96651bffadbf7e80cae2091b18c2a71a9ef03f WatchSource:0}: Error finding container 9bc2907ad607884790bc84cc1e96651bffadbf7e80cae2091b18c2a71a9ef03f: Status 404 returned error can't find the container with id 9bc2907ad607884790bc84cc1e96651bffadbf7e80cae2091b18c2a71a9ef03f
Jul 10 19:28:54.734840 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:54.734809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qxct/must-gather-cr5pn" event={"ID":"2ee00290-f312-4760-81e7-cc3a818c0c6a","Type":"ContainerStarted","Data":"9bc2907ad607884790bc84cc1e96651bffadbf7e80cae2091b18c2a71a9ef03f"}
Jul 10 19:28:55.743921 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:55.743882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qxct/must-gather-cr5pn" event={"ID":"2ee00290-f312-4760-81e7-cc3a818c0c6a","Type":"ContainerStarted","Data":"3755356ec4251940e5287aa908a8efa12880575f497f87ae5824c1a09007f85b"}
Jul 10 19:28:55.744479 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:55.744445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qxct/must-gather-cr5pn" event={"ID":"2ee00290-f312-4760-81e7-cc3a818c0c6a","Type":"ContainerStarted","Data":"199e752826d2d4f4ef476aaf7e5762097315cba2395508f609451a89c3079367"}
Jul 10 19:28:55.760988 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:55.760910    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9qxct/must-gather-cr5pn" podStartSLOduration=1.926888472 podStartE2EDuration="2.760891164s" totalImagesPullingTime="834.002692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:28:55.759026659 +0000 UTC m=+2917.057368729" watchObservedRunningTime="2026-07-10 19:28:55.760891164 +0000 UTC m=+2917.059233233"
Jul 10 19:28:56.458119 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:56.458082    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2m65d_d0e8e075-a99c-4edf-b1ba-3a5504c6734d/global-pull-secret-syncer/0.log"
Jul 10 19:28:56.620185 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:56.620122    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log"
Jul 10 19:28:56.725251 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:56.725167    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-73.ec2.internal_1dddbeb549db7d5e9c6f115eb82341f0/haproxy/0.log"
Jul 10 19:28:58.677984 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:58.677928    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-5gnjq/must-gather-zqx5d"]
Jul 10 19:28:58.679183 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:58.679124    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-5gnjq/must-gather-zqx5d" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerName="copy" containerID="cri-o://2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d" gracePeriod=2
Jul 10 19:28:58.681667 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:58.681628    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zqx5d\" is forbidden: User \"system:node:ip-10-0-129-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5gnjq\": no relationship found between node 'ip-10-0-129-73.ec2.internal' and this object" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:58.685210 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:58.685187    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-5gnjq/must-gather-zqx5d"]
Jul 10 19:28:59.029311 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.029047    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5gnjq_must-gather-zqx5d_3fe7cbb1-4462-4886-aebb-3c774e8dd08f/copy/0.log"
Jul 10 19:28:59.032508 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.032029    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:59.036975 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:59.035175    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zqx5d\" is forbidden: User \"system:node:ip-10-0-129-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5gnjq\": no relationship found between node 'ip-10-0-129-73.ec2.internal' and this object" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:59.123775 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.123732    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-must-gather-output\") pod \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\" (UID: \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\") "
Jul 10 19:28:59.124020 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.123815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk8rh\" (UniqueName: \"kubernetes.io/projected/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-kube-api-access-zk8rh\") pod \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\" (UID: \"3fe7cbb1-4462-4886-aebb-3c774e8dd08f\") "
Jul 10 19:28:59.126661 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.126630    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "3fe7cbb1-4462-4886-aebb-3c774e8dd08f" (UID: "3fe7cbb1-4462-4886-aebb-3c774e8dd08f"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:28:59.127390 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.127360    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-kube-api-access-zk8rh" (OuterVolumeSpecName: "kube-api-access-zk8rh") pod "3fe7cbb1-4462-4886-aebb-3c774e8dd08f" (UID: "3fe7cbb1-4462-4886-aebb-3c774e8dd08f"). InnerVolumeSpecName "kube-api-access-zk8rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:28:59.224925 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.224879    2565 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-must-gather-output\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:59.224925 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.224924    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8rh\" (UniqueName: \"kubernetes.io/projected/3fe7cbb1-4462-4886-aebb-3c774e8dd08f-kube-api-access-zk8rh\") on node \"ip-10-0-129-73.ec2.internal\" DevicePath \"\""
Jul 10 19:28:59.272758 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.272673    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:59.273212 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:59.273174    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zqx5d\" is forbidden: User \"system:node:ip-10-0-129-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5gnjq\": no relationship found between node 'ip-10-0-129-73.ec2.internal' and this object" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:59.282107 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.282069    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" path="/var/lib/kubelet/pods/3fe7cbb1-4462-4886-aebb-3c774e8dd08f/volumes"
Jul 10 19:28:59.372704 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.372562    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cjfx_cf5cc605-07e4-464f-a084-1c2f49ae6ebe/konnectivity-agent/0.log" limit=1024
Jul 10 19:28:59.772696 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.772604    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5gnjq_must-gather-zqx5d_3fe7cbb1-4462-4886-aebb-3c774e8dd08f/copy/0.log"
Jul 10 19:28:59.774435 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.773648    2565 generic.go:356] "Generic (PLEG): container finished" podID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerID="2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d" exitCode=143
Jul 10 19:28:59.774435 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.773807    2565 scope.go:117] "RemoveContainer" containerID="2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d"
Jul 10 19:28:59.774435 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.773995    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gnjq/must-gather-zqx5d"
Jul 10 19:28:59.787787 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.787763    2565 scope.go:117] "RemoveContainer" containerID="b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542"
Jul 10 19:28:59.814310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.813605    2565 scope.go:117] "RemoveContainer" containerID="2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d"
Jul 10 19:28:59.814310 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:59.813946    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d\": container with ID starting with 2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d not found: ID does not exist" containerID="2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d"
Jul 10 19:28:59.814310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.813999    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d"} err="failed to get container status \"2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d\": rpc error: code = NotFound desc = could not find container \"2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d\": container with ID starting with 2164f7cedf5fb00e5aa32b4b911e990f430c84e3a972cab0fd0426684cd5460d not found: ID does not exist"
Jul 10 19:28:59.814310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.814024    2565 scope.go:117] "RemoveContainer" containerID="b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542"
Jul 10 19:28:59.814310 ip-10-0-129-73 kubenswrapper[2565]: E0710 19:28:59.814245    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542\": container with ID starting with b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542 not found: ID does not exist" containerID="b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542"
Jul 10 19:28:59.814310 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:28:59.814265    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542"} err="failed to get container status \"b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542\": rpc error: code = NotFound desc = could not find container \"b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542\": container with ID starting with b20d202575f69a7125f6c4d663107a62307a3c060f55b142c6c5cc29afb44542 not found: ID does not exist"
Jul 10 19:29:01.023017 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.022902    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bhp9q_7d68bbe0-dc50-45bc-9176-2603887ce446/kube-state-metrics/0.log"
Jul 10 19:29:01.050033 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.049929    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bhp9q_7d68bbe0-dc50-45bc-9176-2603887ce446/kube-rbac-proxy-main/0.log"
Jul 10 19:29:01.081423 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.081393    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bhp9q_7d68bbe0-dc50-45bc-9176-2603887ce446/kube-rbac-proxy-self/0.log"
Jul 10 19:29:01.166362 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.166233    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-x96hd_223dc601-b63a-4b63-ae11-091f6ae6f6a7/monitoring-plugin/0.log"
Jul 10 19:29:01.380730 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.380706    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvdjd_eacdb005-b9a5-41af-a3a5-25df81aa41a3/node-exporter/0.log"
Jul 10 19:29:01.412730 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.412700    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvdjd_eacdb005-b9a5-41af-a3a5-25df81aa41a3/kube-rbac-proxy/0.log"
Jul 10 19:29:01.438435 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.438402    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvdjd_eacdb005-b9a5-41af-a3a5-25df81aa41a3/init-textfile/0.log"
Jul 10 19:29:01.477506 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.477473    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n8xrf_7dccc0b5-9d9e-46c4-8255-5c874c289bfc/kube-rbac-proxy-main/0.log"
Jul 10 19:29:01.503004 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.502971    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n8xrf_7dccc0b5-9d9e-46c4-8255-5c874c289bfc/kube-rbac-proxy-self/0.log"
Jul 10 19:29:01.534815 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.534782    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n8xrf_7dccc0b5-9d9e-46c4-8255-5c874c289bfc/openshift-state-metrics/0.log"
Jul 10 19:29:01.586214 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.586132    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_229b05ad-2c70-4050-ac5c-9976a0e27f8b/prometheus/0.log"
Jul 10 19:29:01.608383 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.608350    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_229b05ad-2c70-4050-ac5c-9976a0e27f8b/config-reloader/0.log"
Jul 10 19:29:01.635059 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.635025    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_229b05ad-2c70-4050-ac5c-9976a0e27f8b/thanos-sidecar/0.log"
Jul 10 19:29:01.664746 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.664715    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_229b05ad-2c70-4050-ac5c-9976a0e27f8b/kube-rbac-proxy-web/0.log"
Jul 10 19:29:01.695790 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.695757    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_229b05ad-2c70-4050-ac5c-9976a0e27f8b/kube-rbac-proxy/0.log"
Jul 10 19:29:01.721683 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.721652    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_229b05ad-2c70-4050-ac5c-9976a0e27f8b/kube-rbac-proxy-thanos/0.log"
Jul 10 19:29:01.748182 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.748151    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_229b05ad-2c70-4050-ac5c-9976a0e27f8b/init-config-reloader/0.log"
Jul 10 19:29:01.788469 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.788436    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-8jxkm_b132a3f7-4f59-4159-9411-8e3562002c89/prometheus-operator/0.log"
Jul 10 19:29:01.808147 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.808114    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-8jxkm_b132a3f7-4f59-4159-9411-8e3562002c89/kube-rbac-proxy/0.log"
Jul 10 19:29:01.961479 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.961444    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/thanos-query/0.log"
Jul 10 19:29:01.986083 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:01.986051    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/kube-rbac-proxy-web/0.log"
Jul 10 19:29:02.018764 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:02.018734    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/kube-rbac-proxy/0.log"
Jul 10 19:29:02.046000 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:02.045947    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/prom-label-proxy/0.log"
Jul 10 19:29:02.072031 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:02.071998    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/kube-rbac-proxy-rules/0.log"
Jul 10 19:29:02.097876 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:02.097844    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdbbc5c54-7hgkl_70b9e1de-ad0c-40e7-98af-d4664074ca21/kube-rbac-proxy-metrics/0.log"
Jul 10 19:29:04.430630 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.430585    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-x8cj5_a51a24dc-e29e-4dd7-a621-4c169d6d282d/download-server/0.log"
Jul 10 19:29:04.923651 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.923612    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"]
Jul 10 19:29:04.924144 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.924116    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerName="gather"
Jul 10 19:29:04.924144 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.924145    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerName="gather"
Jul 10 19:29:04.924316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.924172    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerName="copy"
Jul 10 19:29:04.924316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.924180    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerName="copy"
Jul 10 19:29:04.924316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.924245    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerName="gather"
Jul 10 19:29:04.924316 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.924256    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe7cbb1-4462-4886-aebb-3c774e8dd08f" containerName="copy"
Jul 10 19:29:04.929453 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.929428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:04.937327 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.937302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"]
Jul 10 19:29:04.995125 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.995081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-proc\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:04.995334 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.995173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-sys\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:04.995334 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.995231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-podres\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:04.995334 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.995285    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-lib-modules\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:04.995334 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:04.995317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npggg\" (UniqueName: \"kubernetes.io/projected/79991409-78f1-4565-8702-b723e792542a-kube-api-access-npggg\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096117 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-lib-modules\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096117 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npggg\" (UniqueName: \"kubernetes.io/projected/79991409-78f1-4565-8702-b723e792542a-kube-api-access-npggg\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-proc\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-sys\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-podres\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-lib-modules\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-proc\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-sys\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.096360 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.096340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79991409-78f1-4565-8702-b723e792542a-podres\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.105106 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.105079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npggg\" (UniqueName: \"kubernetes.io/projected/79991409-78f1-4565-8702-b723e792542a-kube-api-access-npggg\") pod \"perf-node-gather-daemonset-2wb4d\" (UID: \"79991409-78f1-4565-8702-b723e792542a\") " pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.241342 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.241197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.412887 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.412855    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"]
Jul 10 19:29:05.418228 ip-10-0-129-73 kubenswrapper[2565]: W0710 19:29:05.416749    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79991409_78f1_4565_8702_b723e792542a.slice/crio-089cb2d847eefbfc321194c3591931f88722ddb76f156a38aa306aa929d93be3 WatchSource:0}: Error finding container 089cb2d847eefbfc321194c3591931f88722ddb76f156a38aa306aa929d93be3: Status 404 returned error can't find the container with id 089cb2d847eefbfc321194c3591931f88722ddb76f156a38aa306aa929d93be3
Jul 10 19:29:05.816002 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.815883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d" event={"ID":"79991409-78f1-4565-8702-b723e792542a","Type":"ContainerStarted","Data":"a3d4b97d1b8c6013979de52d85c9e6160839ee8c62f71b629d9d6db2e5bf8aa0"}
Jul 10 19:29:05.816002 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.815980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d" event={"ID":"79991409-78f1-4565-8702-b723e792542a","Type":"ContainerStarted","Data":"089cb2d847eefbfc321194c3591931f88722ddb76f156a38aa306aa929d93be3"}
Jul 10 19:29:05.816740 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.816714    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:05.833655 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.833611    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d" podStartSLOduration=1.83359694 podStartE2EDuration="1.83359694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:29:05.832462294 +0000 UTC m=+2927.130804363" watchObservedRunningTime="2026-07-10 19:29:05.83359694 +0000 UTC m=+2927.131939009"
Jul 10 19:29:05.968235 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.968149    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z2pk2_c4f27378-5bf0-4d9a-9508-06c76b584cdc/dns/0.log"
Jul 10 19:29:05.992563 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:05.992519    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z2pk2_c4f27378-5bf0-4d9a-9508-06c76b584cdc/kube-rbac-proxy/0.log"
Jul 10 19:29:06.066457 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:06.066342    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wjcdv_fb2831cc-2294-4c4e-9cb4-cee0fbc3a2f6/dns-node-resolver/0.log"
Jul 10 19:29:06.579907 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:06.579875    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qp9st_e2e93258-74e9-4eb4-b466-b3a5141554a2/node-ca/0.log"
Jul 10 19:29:07.757852 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:07.757818    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nxzsc_fd2847f5-9eae-4319-a9e4-061d9d3f2da1/serve-healthcheck-canary/0.log"
Jul 10 19:29:08.195659 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:08.195632    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5f954_c02d7e07-698b-4351-9db2-0b570b3b8570/kube-rbac-proxy/0.log"
Jul 10 19:29:08.216294 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:08.216263    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5f954_c02d7e07-698b-4351-9db2-0b570b3b8570/exporter/0.log"
Jul 10 19:29:08.237587 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:08.237547    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5f954_c02d7e07-698b-4351-9db2-0b570b3b8570/extractor/0.log"
Jul 10 19:29:10.541916 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:10.541887    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2kl9d_3c1cd301-000e-47ff-bf64-4588692e7cf7/s3-init/0.log"
Jul 10 19:29:10.573102 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:10.573059    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qbzzv_535af7ea-87c5-4685-b6b2-dc12bd2e77f6/seaweedfs/0.log"
Jul 10 19:29:12.840595 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:12.839871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9qxct/perf-node-gather-daemonset-2wb4d"
Jul 10 19:29:15.766440 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:15.766381    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6x46v_28fbcd0b-2152-4351-9455-60a660f1e83a/kube-multus/0.log"
Jul 10 19:29:15.791665 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:15.791637    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lrrk_4cd99848-3cfb-480a-8c22-b155ccb73066/kube-multus-additional-cni-plugins/0.log"
Jul 10 19:29:15.815542 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:15.815518    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lrrk_4cd99848-3cfb-480a-8c22-b155ccb73066/egress-router-binary-copy/0.log"
Jul 10 19:29:15.840538 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:15.840497    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lrrk_4cd99848-3cfb-480a-8c22-b155ccb73066/cni-plugins/0.log"
Jul 10 19:29:15.862228 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:15.862201    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lrrk_4cd99848-3cfb-480a-8c22-b155ccb73066/bond-cni-plugin/0.log"
Jul 10 19:29:15.886900 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:15.886871    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lrrk_4cd99848-3cfb-480a-8c22-b155ccb73066/routeoverride-cni/0.log"
Jul 10 19:29:15.913845 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:15.913812    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lrrk_4cd99848-3cfb-480a-8c22-b155ccb73066/whereabouts-cni-bincopy/0.log"
Jul 10 19:29:15.937270 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:15.937244    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lrrk_4cd99848-3cfb-480a-8c22-b155ccb73066/whereabouts-cni/0.log"
Jul 10 19:29:16.411945 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:16.411915    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vhp8k_f1100f38-e07c-440d-b610-7f2dc8035c9c/network-metrics-daemon/0.log"
Jul 10 19:29:16.429578 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:16.429549    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vhp8k_f1100f38-e07c-440d-b610-7f2dc8035c9c/kube-rbac-proxy/0.log"
Jul 10 19:29:17.232835 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.232798    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-controller/0.log"
Jul 10 19:29:17.252382 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.252348    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/1.log"
Jul 10 19:29:17.252550 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.252410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovn-acl-logging/0.log"
Jul 10 19:29:17.275049 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.275010    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/kube-rbac-proxy-node/0.log"
Jul 10 19:29:17.305505 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.305479    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/kube-rbac-proxy-ovn-metrics/0.log"
Jul 10 19:29:17.330247 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.330204    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/northd/0.log"
Jul 10 19:29:17.357633 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.357603    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/nbdb/0.log"
Jul 10 19:29:17.380817 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.380784    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/sbdb/0.log"
Jul 10 19:29:17.458860 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:17.458828    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l79f7_a206ccd3-28b2-41c5-acb7-833d43e07bd3/ovnkube-controller/0.log"
Jul 10 19:29:18.987178 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:18.987135    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vfbvz_3be1ba28-5b3d-4f64-9575-5e98a563732d/network-check-target-container/0.log"
Jul 10 19:29:19.911161 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:19.911082    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-m7t2d_6a4647ef-1fd0-4d22-9037-298a644990a6/iptables-alerter/0.log"
Jul 10 19:29:20.584498 ip-10-0-129-73 kubenswrapper[2565]: I0710 19:29:20.584463    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-t629v_be89d687-cb34-467b-b5c3-73206230e2ab/tuned/0.log"