Jul 07 11:40:13.660931 ip-10-0-140-177 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.104958    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110507    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110521    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110525    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110528    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110531    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110534    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110537    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.163939 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110539    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110542    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110544    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110547    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110549    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110552    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110554    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110557    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110560    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110563    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110566    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110569    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110571    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110573    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110576    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110578    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110581    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110584    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110594    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110597    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.164791 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110600    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110602    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110605    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110607    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110610    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110612    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110615    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110618    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110620    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110623    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110625    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110628    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110630    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110632    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110635    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110637    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110640    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110643    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110645    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110647    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.165433 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110650    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110674    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110680    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110683    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110687    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110690    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110693    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110696    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110699    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110701    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110704    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110707    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110711    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110713    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110716    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110719    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110723    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110727    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110731    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.166093 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110735    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110738    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110740    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110743    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110745    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110748    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110750    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110753    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110755    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110758    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110760    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110763    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110765    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110768    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110771    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110774    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110776    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110780    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110783    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110785    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.166749 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110788    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110791    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110793    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110796    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110798    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110801    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110803    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110806    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110808    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110811    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110814    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110816    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.110819    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111499    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111506    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111509    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111514    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111518    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111521    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.167435 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111524    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111527    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111530    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111532    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111535    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111538    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111541    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111544    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111547    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111550    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111552    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111555    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111558    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111560    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111563    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111566    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111569    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111572    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111575    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.168265 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111577    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111580    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111582    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111585    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111588    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111590    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111593    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111595    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111598    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111600    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111603    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111606    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111608    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111610    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111613    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111615    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111618    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111621    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111623    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111626    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.169092 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111628    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111631    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111633    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111636    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111638    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111642    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111644    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111647    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111650    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111652    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111675    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111678    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111680    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111683    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111685    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111688    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111690    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111693    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111696    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111698    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.169919 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111701    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111703    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111706    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111708    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111711    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111713    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111719    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111723    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111726    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111729    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111732    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111735    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111737    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111740    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111742    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111745    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111747    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111750    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111754    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.170545 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111758    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111760    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111763    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111766    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111768    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111771    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111773    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111775    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111778    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111780    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111783    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111785    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111788    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111791    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.111794    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112151    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112155    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112158    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112161    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.171622 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112163    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112166    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112169    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112171    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112174    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112176    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112179    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112182    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112186    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112188    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112191    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112194    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112199    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112208    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112211    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112214    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112217    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112219    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112222    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112225    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.172673 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112228    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112231    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112234    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112237    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112240    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112243    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112246    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112249    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112252    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112254    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112257    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112259    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112261    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112264    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112266    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112269    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112271    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112274    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112276    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.173627 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112279    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112281    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112284    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112286    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112289    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112291    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112296    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112298    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112301    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112303    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112306    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112308    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112312    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112314    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112317    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112319    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112322    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112324    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112327    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.174395 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112337    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112341    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112343    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112346    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112348    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112351    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112353    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112356    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112359    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112362    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112365    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112368    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112370    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112373    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112375    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112378    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112380    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112384    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112388    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112392    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.175369 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112396    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.176061 ip-10-0-140-177 systemd[1]: Started Kubernetes Kubelet.
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112399    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112401    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112405    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112408    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112411    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112413    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112416    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112418    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112421    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112423    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112426    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112428    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112431    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112435    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112438    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.112441    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112505    2572 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112516    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112523    2572 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112527    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 11:40:14.176351 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112531    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112534    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112538    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112542    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112545    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112547    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112551    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112554    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112557    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112560    2572 flags.go:64] FLAG: --cgroup-root=""
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112564    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112567    2572 flags.go:64] FLAG: --client-ca-file=""
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112570    2572 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112576    2572 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112602    2572 flags.go:64] FLAG: --cluster-domain=""
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112605    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112609    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112612    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112615    2572 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112619    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112622    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112625    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112629    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112632    2572 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 11:40:14.177345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112635    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112638    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112641    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112643    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112647    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112650    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112653    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112670    2572 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112673    2572 flags.go:64] FLAG: --enable-server="true"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112676    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112680    2572 flags.go:64] FLAG: --event-burst="100"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112683    2572 flags.go:64] FLAG: --event-qps="50"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112686    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112689    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112691    2572 flags.go:64] FLAG: --eviction-hard=""
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112695    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112698    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112701    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112705    2572 flags.go:64] FLAG: --eviction-soft=""
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112708    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112710    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112717    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112720    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112723    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112726    2572 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 11:40:14.178433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112729    2572 flags.go:64] FLAG: --feature-gates=""
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112733    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112736    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112739    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112742    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112745    2572 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112748    2572 flags.go:64] FLAG: --help="false"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112750    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112754    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112756    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112759    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112762    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112766    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112769    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112772    2572 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112774    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112777    2572 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112780    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112783    2572 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112786    2572 flags.go:64] FLAG: --kube-reserved=""
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112789    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112791    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112794    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112797    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 11:40:14.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112799    2572 flags.go:64] FLAG: --lock-file=""
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112804    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112807    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112809    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112815    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112819    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112822    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112825    2572 flags.go:64] FLAG: --logging-format="text"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112828    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112831    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112834    2572 flags.go:64] FLAG: --manifest-url=""
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112837    2572 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112842    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112845    2572 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112848    2572 flags.go:64] FLAG: --max-pods="110"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112851    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112854    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112857    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112860    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112863    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112866    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112868    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112875    2572 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112878    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112881    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 11:40:14.183088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112884    2572 flags.go:64] FLAG: --pod-cidr=""
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112887    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112892    2572 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112895    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112898    2572 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112900    2572 flags.go:64] FLAG: --port="10250"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112904    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112906    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b0ea584290fbbd74"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112911    2572 flags.go:64] FLAG: --qos-reserved=""
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112914    2572 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112916    2572 flags.go:64] FLAG: --register-node="true"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112919    2572 flags.go:64] FLAG: --register-with-taints=""
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112923    2572 flags.go:64] FLAG: --registry-burst="10"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112927    2572 flags.go:64] FLAG: --registry-qps="5"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112930    2572 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112932    2572 flags.go:64] FLAG: --reserved-memory=""
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112936    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112939    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112942    2572 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112945    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112948    2572 flags.go:64] FLAG: --runonce="false"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112950    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112953    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112956    2572 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112959    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 11:40:14.183801 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112962    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112965    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112968    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112971    2572 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112973    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112976    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112979    2572 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112981    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112984    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112987    2572 flags.go:64] FLAG: --system-cgroups=""
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112990    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112995    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.112998    2572 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113001    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113005    2572 flags.go:64] FLAG: --tls-min-version=""
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113009    2572 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113012    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113014    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113017    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113022    2572 flags.go:64] FLAG: --v="2"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113026    2572 flags.go:64] FLAG: --version="false"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113030    2572 flags.go:64] FLAG: --vmodule=""
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113034    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.113037    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113144    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.312269 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113148    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113151    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113154    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113157    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113160    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113163    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113166    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113169    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113171    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113174    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113176    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113179    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113181    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113184    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113186    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113189    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113191    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113194    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113196    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113199    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.336385 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113201    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113204    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113207    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113210    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113212    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113215    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113219    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113222    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113224    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113227    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113230    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113233    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113235    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113238    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113240    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113243    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113245    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113248    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113250    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113253    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.337651 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113255    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113257    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113260    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113263    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113265    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113268    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113272    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113275    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113278    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113280    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113283    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113285    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113288    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113290    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113294    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113296    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113299    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113302    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113305    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.339049 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113308    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113310    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113313    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113317    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113321    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113324    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113327    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113329    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113332    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113335    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113338    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113341    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113344    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113347    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113350    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113352    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113355    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113357    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113359    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.340088 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113362    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113364    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113367    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113369    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113372    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113374    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113377    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113380    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113385    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113388    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113391    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113393    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113397    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113400    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113403    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113406    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113408    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113411    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113414    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.341170 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.113417    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.114289    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.120274    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.120290    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.120309    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.120314    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120389    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120394    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120398    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120402    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120405    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120408    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120411    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120414    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120417    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.342239 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120420    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120423    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120425    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120428    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120432    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120434    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120437    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120439    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120442    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120445    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120448    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120453    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120456    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120458    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120461    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120464    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120466    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120469    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120471    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120474    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.343125 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120477    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120479    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120482    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120484    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120487    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120490    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120492    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120495    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120497    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120500    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120502    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120505    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120509    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120512    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120514    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120518    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120522    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120525    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120528    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.343997 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120530    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120533    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120535    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120538    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120541    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120544    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120548    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120551    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120555    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120557    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120560    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120563    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120566    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120568    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120571    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120574    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120576    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120579    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120581    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120584    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.344875 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120586    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120589    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120591    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120594    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120596    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120600    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120603    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120606    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120608    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120611    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120613    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120616    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120618    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120621    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120624    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120626    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120630    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120632    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120635    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120637    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.345944 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120640    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120642    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120646    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120649    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120652    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120670    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120674    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120677    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120680    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120682    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120685    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.120690    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120794    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120798    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.346817 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120801    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120804    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120807    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120809    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120814    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120817    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120819    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120822    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120825    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120829    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120833    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120836    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120839    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120842    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120846    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120848    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120851    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120853    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120855    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120858    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.347458 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120860    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120863    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120865    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120868    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120871    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120873    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120876    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120878    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120881    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120883    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120886    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120888    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120891    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120893    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120896    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120898    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120902    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120905    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120907    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.348339 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120910    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120912    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120915    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120917    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120920    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120923    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120925    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120930    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120932    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120935    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120938    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120940    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120943    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120945    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120948    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120950    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120952    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120955    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120957    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120960    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.349198 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120963    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120966    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120968    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120971    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120973    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120975    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120978    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120980    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120983    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120987    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120990    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120992    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120995    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.120997    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121000    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121002    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121005    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121007    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121010    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121013    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.350064 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121015    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121018    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121020    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121023    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121025    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121028    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121030    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121033    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121037    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121040    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121043    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121046    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121049    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121052    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121055    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121059    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121063    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.350943 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:14.121066    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.121071    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.121160    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.122948    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.124247    2572 server.go:1019] "Starting client certificate rotation"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.124348    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.124393    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.126421    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.130960    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.140769    2572 log.go:25] "Validated CRI v1 runtime API"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.143612    2572 log.go:25] "Validated CRI v1 image API"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.144421    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 11:40:14.352010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.150057    2572 fs.go:135] Filesystem UUIDs: map[4ed616df-6c65-4e7e-9401-e77d01ebc434:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bafa84b5-3b0b-4325-ad94-9c40b716220d:/dev/nvme0n1p3]
Jul 07 11:40:14.352682 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.150071    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 11:40:14.352883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.155075    2572 manager.go:216] Machine: {Timestamp:2026-07-07 11:40:14.153979582 +0000 UTC m=+0.349556594 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3158801 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e9d8523c868d2bdacc598dcd82a12 SystemUUID:ec2e9d85-23c8-68d2-bdac-c598dcd82a12 BootID:2ee18b23-3ec3-4ac1-a94b-3400d73ab3e5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:c0:96:76:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:c0:96:76:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:b6:b2:0c:fa:d1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 11:40:14.352883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.155175    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 11:40:14.352883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.155244    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 11:40:14.352883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.156134    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 11:40:14.352883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.156153    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 11:40:14.352883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.157297    2572 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 11:40:14.352883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.157314    2572 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 11:40:14.352883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.157323    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.157341    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.157403    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.158313    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.162727    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.162828    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.162856    2572 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.162865    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.162875    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.162884    2572 kubelet.go:398] "Adding apiserver pod source"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.162894    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.163178    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.163197    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.166380    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.168442    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.168484    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169438    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169458    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169468    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169476    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169490    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169499    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169510    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169518    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169528    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169538    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169557    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.169565    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.170320    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.170333    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.174650    2572 server.go:1268] "Started kubelet"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.174759    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.175510    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.175689    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.187076    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.187191    2572 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.187350    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.187364    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-177.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.187566    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 11:40:14.355692 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.188943    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.189779    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-177.ec2.internal.18bfff0e42a31b86  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-177.ec2.internal,UID:ip-10-0-140-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-177.ec2.internal,},FirstTimestamp:2026-07-07 11:40:14.174616454 +0000 UTC m=+0.370193472,LastTimestamp:2026-07-07 11:40:14.174616454 +0000 UTC m=+0.370193472,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-177.ec2.internal,}"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.191198    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9xz7j"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.191976    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.191998    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.192327    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.192702    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.192706    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.192727    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.192828    2572 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.192838    2572 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.203769    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9xz7j"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.206301    2572 factory.go:55] Registering systemd factory
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.206340    2572 factory.go:223] Registration of the systemd container factory successfully
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.206592    2572 factory.go:153] Registering CRI-O factory
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.206606    2572 factory.go:223] Registration of the crio container factory successfully
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.206693    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.206714    2572 factory.go:103] Registering Raw factory
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.206724    2572 manager.go:1196] Started watching for new ooms in manager
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.207993    2572 manager.go:349] Starting recovery of all containers
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.210621    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.210835    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.218923    2572 manager.go:354] Recovery completed
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.222895    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.229804    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.229828    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.229840    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.230302    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.230314    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.230316    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.230332    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.231046    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.357421 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.231069    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.231078    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.232385    2572 policy_none.go:49] "None policy: Start"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.232397    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.232419    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.234410    2572 policy_none.go:47] "Start"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.282180    2572 manager.go:337] "Starting Device Plugin manager"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.282205    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.282213    2572 server.go:87] "Starting device plugin registration server"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.282386    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.282397    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.282491    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.282572    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.282580    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.283030    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.283069    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.382860    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.383683    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.383706    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.383718    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.383741    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.390410    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.390428    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-177.ec2.internal\": node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.422497    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.431690    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.432858    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.432896    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.432913    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.432942    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 11:40:14.496354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.435259    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.523372    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.533218    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal"]
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.533883    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.534645    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.534685    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.534700    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.536872    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.536999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.537024    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.537613    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.537633    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.537643    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.537967    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.537995    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.538006    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.539824    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.539847    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.540564    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.540586    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.540600    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.569250    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-177.ec2.internal\" not found" node="ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.573088    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-177.ec2.internal\" not found" node="ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.595475    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46f616d7126e59bf8c9ddddc795b4398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal\" (UID: \"46f616d7126e59bf8c9ddddc795b4398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.595511    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f616d7126e59bf8c9ddddc795b4398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal\" (UID: \"46f616d7126e59bf8c9ddddc795b4398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.595533    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf25b0953b6a4731ed485a31472da6c0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-177.ec2.internal\" (UID: \"bf25b0953b6a4731ed485a31472da6c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.650536 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.623598    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:14.782165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.696259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f616d7126e59bf8c9ddddc795b4398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal\" (UID: \"46f616d7126e59bf8c9ddddc795b4398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.782165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.696287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf25b0953b6a4731ed485a31472da6c0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-177.ec2.internal\" (UID: \"bf25b0953b6a4731ed485a31472da6c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.782165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.696304    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46f616d7126e59bf8c9ddddc795b4398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal\" (UID: \"46f616d7126e59bf8c9ddddc795b4398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.782165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.696352    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46f616d7126e59bf8c9ddddc795b4398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal\" (UID: \"46f616d7126e59bf8c9ddddc795b4398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.782165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.696367    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f616d7126e59bf8c9ddddc795b4398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal\" (UID: \"46f616d7126e59bf8c9ddddc795b4398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.782165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.696381    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf25b0953b6a4731ed485a31472da6c0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-177.ec2.internal\" (UID: \"bf25b0953b6a4731ed485a31472da6c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.782165 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.725564    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:14.914495 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.827368    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:14.914495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.874340    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:14.914495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:14.877311    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:15.030990 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:14.929485    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:15.031210 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:15.031000    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:15.147459 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.124253    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 11:40:15.147459 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.124366    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 11:40:15.147459 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:15.132046    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:15.269068 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.211814    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 11:35:14 +0000 UTC" deadline="2028-01-09 05:34:23.101827608 +0000 UTC"
Jul 07 11:40:15.269068 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.211849    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13217h54m7.88998188s"
Jul 07 11:40:15.269068 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:15.232956    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:15.388122 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:15.333750    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:15.471901 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:15.434116    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:15.471901 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.435610    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:15.568066 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.524268    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:15.568066 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:15.534800    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-177.ec2.internal\" not found"
Jul 07 11:40:15.678327 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.601823    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:15.759596 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.692401    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:15.759596 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:15.696963    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f616d7126e59bf8c9ddddc795b4398.slice/crio-2df63fc8dff63719bb0750e427556548771019d766227d07df5fe41257145bae WatchSource:0}: Error finding container 2df63fc8dff63719bb0750e427556548771019d766227d07df5fe41257145bae: Status 404 returned error can't find the container with id 2df63fc8dff63719bb0750e427556548771019d766227d07df5fe41257145bae
Jul 07 11:40:15.759596 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:15.697506    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf25b0953b6a4731ed485a31472da6c0.slice/crio-afc961d78ea154a5eb67adcc05ad3a0e134a069b30f110838d9431566d92b5df WatchSource:0}: Error finding container afc961d78ea154a5eb67adcc05ad3a0e134a069b30f110838d9431566d92b5df: Status 404 returned error can't find the container with id afc961d78ea154a5eb67adcc05ad3a0e134a069b30f110838d9431566d92b5df
Jul 07 11:40:15.759596 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.702706    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:40:15.759596 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.706037    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 11:40:15.759596 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.707103    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal"
Jul 07 11:40:15.759596 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:15.738068    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 11:40:16.018952 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.018926    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:16.164889 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.164859    2572 apiserver.go:52] "Watching apiserver"
Jul 07 11:40:16.173140 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.173115    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 11:40:16.173593 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.173565    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-465p6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5","openshift-cluster-node-tuning-operator/tuned-xdn6h","openshift-dns/node-resolver-g2rkx","openshift-image-registry/node-ca-zh7g2","openshift-multus/multus-additional-cni-plugins-nm6fk","openshift-multus/multus-fpndh","kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal","openshift-multus/network-metrics-daemon-nwrkz","openshift-network-diagnostics/network-check-target-zzbqh","openshift-network-operator/iptables-alerter-bb9dr","openshift-ovn-kubernetes/ovnkube-node-hpspv","kube-system/global-pull-secret-syncer-wrrg9"]
Jul 07 11:40:16.176319 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.176295    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.178786 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.178752    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.179495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.179473    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kzmr2\""
Jul 07 11:40:16.179895 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.179874    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 11:40:16.180010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.179890    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 11:40:16.180010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.179936    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 11:40:16.180010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.179881    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 11:40:16.181328 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.181306    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.181431 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.181354    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 11:40:16.181634 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.181606    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 11:40:16.181860 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.181841    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 11:40:16.181943 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.181896    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 11:40:16.181943 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.181935    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-547k9\""
Jul 07 11:40:16.182048 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.181961    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 11:40:16.182048 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.181936    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 11:40:16.183702 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.183648    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 11:40:16.184181 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.184153    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k8kbd\""
Jul 07 11:40:16.184181 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.184160    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 11:40:16.184352 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.184297    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.186706 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.186685    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 11:40:16.186874 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.186852    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xs2b2\""
Jul 07 11:40:16.186944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.186906    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 11:40:16.187543 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.187524    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.190093 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.190073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.191045 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.191020    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 11:40:16.191206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.191186    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 11:40:16.191542 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.191521    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-68d5v\""
Jul 07 11:40:16.191655 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.191636    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 11:40:16.192247 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.192227    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 11:40:16.192336 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.192320    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 11:40:16.192559 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.192524    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rqslb\""
Jul 07 11:40:16.192644 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.192628    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 11:40:16.193304 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.192989    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.193304 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.193109    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:16.195833 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.195474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:16.198235 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.198210    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 11:40:16.199241 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.198974    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 11:40:16.199241 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.199085    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.199241 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.199181    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5n5qp\""
Jul 07 11:40:16.201791 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.201506    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 11:40:16.202328 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.202309    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t7cbk\""
Jul 07 11:40:16.202328 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.202321    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 11:40:16.202473 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.202416    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 11:40:16.203684 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.203636    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:16.203902 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.203731    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:16.206614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.206595    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:16.206786 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.206766    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:16.209021 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209001    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.209547 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209434    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-var-lib-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.209547 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209460    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-kubernetes\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.209547 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209481    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-cni-multus\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.209547 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209498    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-kubelet\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.209547 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209521    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-systemd-units\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.209887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209574    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-ovn\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.209887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-systemd\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.209887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209658    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abea6595-66e7-4e5a-8414-e656fbe0b242-cni-binary-copy\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.209887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209706    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5kv\" (UniqueName: \"kubernetes.io/projected/cf3e2930-9990-4025-a22b-b53af21851e7-kube-api-access-nc5kv\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.209887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209749    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cnibin\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.209887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209774    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-node-log\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.209887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-cni-bin\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.209887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209869    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-os-release\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209902    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-lib-modules\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-host\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.209967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-etc-kubernetes\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210011    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-systemd\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210048    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210080    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-cni-netd\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210086    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210119    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysctl-conf\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210146    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf3e2930-9990-4025-a22b-b53af21851e7-host\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210173    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cad78671-89ed-4a34-877e-7e0448976883-ovn-node-metrics-cert\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210218    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-ovnkube-script-lib\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210244    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-env-overrides\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cf3e2930-9990-4025-a22b-b53af21851e7-serviceca\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210291    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-socket-dir-parent\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-os-release\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210345    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-slash\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-ovnkube-config\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210410    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-sys\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66a5788d-0db4-4335-ae38-7e916eca3b86-dbus\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210463    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-hostroot\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210501    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210553    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-modprobe-d\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210575    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-var-lib-kubelet\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210604    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-conf-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.210956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da24ad97-d836-4d7a-9db2-586d261e3349-etc-tuned\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210690    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da24ad97-d836-4d7a-9db2-586d261e3349-tmp\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210726    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nnwq\" (UniqueName: \"kubernetes.io/projected/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-kube-api-access-5nnwq\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210770    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66a5788d-0db4-4335-ae38-7e916eca3b86-kubelet-config\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210794    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-multus-certs\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210821    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64sxz\" (UniqueName: \"kubernetes.io/projected/71053fbc-7333-4971-9c42-0d84b4fb6a6b-kube-api-access-64sxz\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysconfig\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210871    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-system-cni-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210897    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-cnibin\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210922    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-k8s-cni-cncf-io\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdkxc\" (UniqueName: \"kubernetes.io/projected/cad78671-89ed-4a34-877e-7e0448976883-kube-api-access-bdkxc\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.210981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-netns\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-cni-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211032    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-etc-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.211513 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-log-socket\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211183    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysctl-d\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211532    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-run\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211568    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcc5\" (UniqueName: \"kubernetes.io/projected/da24ad97-d836-4d7a-9db2-586d261e3349-kube-api-access-nmcc5\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211624    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-cni-bin\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-kubelet\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-daemon-config\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211694    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhp5\" (UniqueName: \"kubernetes.io/projected/abea6595-66e7-4e5a-8414-e656fbe0b242-kube-api-access-dzhp5\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211734    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-hosts-file\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-system-cni-dir\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211784    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-run-netns\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.212271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-tmp-dir\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.213072 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.211906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.213567 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.213546    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 11:40:16.213653 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.213620    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 11:40:16.213710 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.213690    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 11:40:16.213887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.213866    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-skxlr\""
Jul 07 11:40:16.239071 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.238822    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9mff8"
Jul 07 11:40:16.252997 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.252975    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9mff8"
Jul 07 11:40:16.293814 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.293723    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 11:40:16.312265 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-var-lib-kubelet\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.312366 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312282    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9602c0b-14d5-4878-9bb3-1b8914003013-host-slash\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.312366 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312310    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-conf-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.312366 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.312366 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da24ad97-d836-4d7a-9db2-586d261e3349-etc-tuned\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.312568 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312362    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-var-lib-kubelet\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.312568 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312382    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da24ad97-d836-4d7a-9db2-586d261e3349-tmp\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.312568 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nnwq\" (UniqueName: \"kubernetes.io/projected/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-kube-api-access-5nnwq\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.312568 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312418    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-conf-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.312568 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66a5788d-0db4-4335-ae38-7e916eca3b86-kubelet-config\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.312568 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312461    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-multus-certs\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.312568 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312530    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66a5788d-0db4-4335-ae38-7e916eca3b86-kubelet-config\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.312917 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312762    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 11:40:16.312917 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64sxz\" (UniqueName: \"kubernetes.io/projected/71053fbc-7333-4971-9c42-0d84b4fb6a6b-kube-api-access-64sxz\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.312917 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312838    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysconfig\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.312917 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.312865    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:16.313164 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313147    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-multus-certs\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.313221 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313186    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-system-cni-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.313272 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313213    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-cnibin\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.313272 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-k8s-cni-cncf-io\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.313369 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313307    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-cnibin\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.313369 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdkxc\" (UniqueName: \"kubernetes.io/projected/cad78671-89ed-4a34-877e-7e0448976883-kube-api-access-bdkxc\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.313369 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313361    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-netns\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.313508 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313388    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-cni-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.313508 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313435    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-netns\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.313508 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313463    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-etc-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.313508 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313496    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-log-socket\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.313823 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313521    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysctl-d\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.313823 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313543    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-run\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.313823 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313566    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcc5\" (UniqueName: \"kubernetes.io/projected/da24ad97-d836-4d7a-9db2-586d261e3349-kube-api-access-nmcc5\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.313823 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313594    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n498b\" (UniqueName: \"kubernetes.io/projected/d622f883-4224-4801-b1d3-2b543c30dc35-kube-api-access-n498b\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-cni-bin\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313939    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-run-k8s-cni-cncf-io\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313948    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-kubelet\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313951    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysconfig\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313979    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-log-socket\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313981    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-daemon-config\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313998    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-kubelet\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314008    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhp5\" (UniqueName: \"kubernetes.io/projected/abea6595-66e7-4e5a-8414-e656fbe0b242-kube-api-access-dzhp5\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314030    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-cni-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314044    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-hosts-file\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-system-cni-dir\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314093    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysctl-d\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314101    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314114    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-system-cni-dir\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314135    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-run\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.313894    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314181    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-hosts-file\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314199    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-system-cni-dir\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314211    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-cni-bin\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314239    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-run-netns\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314303    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.314485 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.314412    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314749    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-daemon-config\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314804    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-etc-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314954    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314952    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.314984    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-socket-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315035    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-run-netns\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315066    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-tmp-dir\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.315116    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret podName:66a5788d-0db4-4335-ae38-7e916eca3b86 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:16.815082703 +0000 UTC m=+3.010659728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret") pod "global-pull-secret-syncer-wrrg9" (UID: "66a5788d-0db4-4335-ae38-7e916eca3b86") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:16.315153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315135    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315200    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-var-lib-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-kubernetes\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315286    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzv62\" (UniqueName: \"kubernetes.io/projected/c9602c0b-14d5-4878-9bb3-1b8914003013-kube-api-access-hzv62\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315317    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-cni-multus\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315323    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-tmp-dir\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-kubelet\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-systemd-units\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315397    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-host-var-lib-cni-multus\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315422    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-ovn\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315436    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-var-lib-openvswitch\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315449    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-systemd\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315483    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-kubernetes\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.315550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315536    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5bb35b90-7072-43a8-81de-f8db577cd975-agent-certs\") pod \"konnectivity-agent-465p6\" (UID: \"5bb35b90-7072-43a8-81de-f8db577cd975\") " pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315562    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abea6595-66e7-4e5a-8414-e656fbe0b242-cni-binary-copy\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5kv\" (UniqueName: \"kubernetes.io/projected/cf3e2930-9990-4025-a22b-b53af21851e7-kube-api-access-nc5kv\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315611    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-ovn\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315616    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cnibin\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315670    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cnibin\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315706    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-node-log\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315724    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71053fbc-7333-4971-9c42-0d84b4fb6a6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-cni-bin\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315756    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-os-release\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315772    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-kubelet\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315814    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-systemd-units\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315817    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-lib-modules\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315848    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-cni-bin\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315850    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-host\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.316174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315888    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-os-release\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315899    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-systemd\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315941    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-host\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.315941    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-node-log\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316002    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9602c0b-14d5-4878-9bb3-1b8914003013-iptables-alerter-script\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-lib-modules\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-etc-kubernetes\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316082    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-etc-kubernetes\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316127    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-systemd\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316181    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-run-systemd\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-cni-netd\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysctl-conf\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316290    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-registration-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.316887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316288    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-device-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316344    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316364    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-cni-netd\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316369    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf3e2930-9990-4025-a22b-b53af21851e7-host\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316381    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abea6595-66e7-4e5a-8414-e656fbe0b242-cni-binary-copy\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316399    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cad78671-89ed-4a34-877e-7e0448976883-ovn-node-metrics-cert\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316432    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf3e2930-9990-4025-a22b-b53af21851e7-host\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316468    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-ovnkube-script-lib\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316497    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-sysctl-conf\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316504    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-env-overrides\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316612    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cf3e2930-9990-4025-a22b-b53af21851e7-serviceca\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316638    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-socket-dir-parent\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316713    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-os-release\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-slash\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.317525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316761    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-ovnkube-config\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316810    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-sys\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5bb35b90-7072-43a8-81de-f8db577cd975-konnectivity-ca\") pod \"konnectivity-agent-465p6\" (UID: \"5bb35b90-7072-43a8-81de-f8db577cd975\") " pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-sys-fs\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316894    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzc85\" (UniqueName: \"kubernetes.io/projected/641d4e40-f66b-40c2-8db4-66b16d6958ad-kube-api-access-nzc85\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316883    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da24ad97-d836-4d7a-9db2-586d261e3349-tmp\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66a5788d-0db4-4335-ae38-7e916eca3b86-dbus\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316945    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-hostroot\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.316992    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-os-release\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317062    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-ovnkube-script-lib\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317104    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-slash\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317146    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-sys\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.318232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-modprobe-d\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317066    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da24ad97-d836-4d7a-9db2-586d261e3349-etc-modprobe-d\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317236    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66a5788d-0db4-4335-ae38-7e916eca3b86-dbus\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317271    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-hostroot\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317451    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-ovnkube-config\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317516    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad78671-89ed-4a34-877e-7e0448976883-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317565    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cad78671-89ed-4a34-877e-7e0448976883-env-overrides\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317619    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abea6595-66e7-4e5a-8414-e656fbe0b242-multus-socket-dir-parent\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317677    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71053fbc-7333-4971-9c42-0d84b4fb6a6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.317682    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cf3e2930-9990-4025-a22b-b53af21851e7-serviceca\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.318891 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.318186    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da24ad97-d836-4d7a-9db2-586d261e3349-etc-tuned\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.319405 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.319383    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cad78671-89ed-4a34-877e-7e0448976883-ovn-node-metrics-cert\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.328635 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.328607    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5kv\" (UniqueName: \"kubernetes.io/projected/cf3e2930-9990-4025-a22b-b53af21851e7-kube-api-access-nc5kv\") pod \"node-ca-zh7g2\" (UID: \"cf3e2930-9990-4025-a22b-b53af21851e7\") " pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.329025 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.328983    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcc5\" (UniqueName: \"kubernetes.io/projected/da24ad97-d836-4d7a-9db2-586d261e3349-kube-api-access-nmcc5\") pod \"tuned-xdn6h\" (UID: \"da24ad97-d836-4d7a-9db2-586d261e3349\") " pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.330182 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.330105    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdkxc\" (UniqueName: \"kubernetes.io/projected/cad78671-89ed-4a34-877e-7e0448976883-kube-api-access-bdkxc\") pod \"ovnkube-node-hpspv\" (UID: \"cad78671-89ed-4a34-877e-7e0448976883\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.331566 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.331534    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64sxz\" (UniqueName: \"kubernetes.io/projected/71053fbc-7333-4971-9c42-0d84b4fb6a6b-kube-api-access-64sxz\") pod \"multus-additional-cni-plugins-nm6fk\" (UID: \"71053fbc-7333-4971-9c42-0d84b4fb6a6b\") " pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.331744 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.331723    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nnwq\" (UniqueName: \"kubernetes.io/projected/5d9ceae5-ab86-4d83-b0ff-c5912fee77cd-kube-api-access-5nnwq\") pod \"node-resolver-g2rkx\" (UID: \"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd\") " pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.332689 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.332633    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhp5\" (UniqueName: \"kubernetes.io/projected/abea6595-66e7-4e5a-8414-e656fbe0b242-kube-api-access-dzhp5\") pod \"multus-fpndh\" (UID: \"abea6595-66e7-4e5a-8414-e656fbe0b242\") " pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.418452 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418424    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5bb35b90-7072-43a8-81de-f8db577cd975-konnectivity-ca\") pod \"konnectivity-agent-465p6\" (UID: \"5bb35b90-7072-43a8-81de-f8db577cd975\") " pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:16.418452 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418455    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-sys-fs\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.418634 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzc85\" (UniqueName: \"kubernetes.io/projected/641d4e40-f66b-40c2-8db4-66b16d6958ad-kube-api-access-nzc85\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.418634 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418504    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9602c0b-14d5-4878-9bb3-1b8914003013-host-slash\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.418634 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:16.418634 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418597    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n498b\" (UniqueName: \"kubernetes.io/projected/d622f883-4224-4801-b1d3-2b543c30dc35-kube-api-access-n498b\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:16.418634 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:16.418948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.418948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418749    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-socket-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.418948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzv62\" (UniqueName: \"kubernetes.io/projected/c9602c0b-14d5-4878-9bb3-1b8914003013-kube-api-access-hzv62\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.418948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5bb35b90-7072-43a8-81de-f8db577cd975-agent-certs\") pod \"konnectivity-agent-465p6\" (UID: \"5bb35b90-7072-43a8-81de-f8db577cd975\") " pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:16.418948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418841    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9602c0b-14d5-4878-9bb3-1b8914003013-iptables-alerter-script\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.418948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418869    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-registration-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.418948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-device-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.418948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.419314 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.418996    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5bb35b90-7072-43a8-81de-f8db577cd975-konnectivity-ca\") pod \"konnectivity-agent-465p6\" (UID: \"5bb35b90-7072-43a8-81de-f8db577cd975\") " pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:16.419314 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.419056    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.419314 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.419079    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.419314 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.419104    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-sys-fs\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.419314 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.419191    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-socket-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.419538 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.419339    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9602c0b-14d5-4878-9bb3-1b8914003013-host-slash\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.419538 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.419419    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:16.419538 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.419477    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs podName:d622f883-4224-4801-b1d3-2b543c30dc35 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:16.919461613 +0000 UTC m=+3.115038613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs") pod "network-metrics-daemon-nwrkz" (UID: "d622f883-4224-4801-b1d3-2b543c30dc35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:16.419999 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.419977    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-registration-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.420508 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.420481    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9602c0b-14d5-4878-9bb3-1b8914003013-iptables-alerter-script\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.420575 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.420554    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/641d4e40-f66b-40c2-8db4-66b16d6958ad-device-dir\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.422942 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.422922    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5bb35b90-7072-43a8-81de-f8db577cd975-agent-certs\") pod \"konnectivity-agent-465p6\" (UID: \"5bb35b90-7072-43a8-81de-f8db577cd975\") " pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:16.435569 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.435549    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:16.435651 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.435573    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:16.435651 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.435586    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fm8rq for pod openshift-network-diagnostics/network-check-target-zzbqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:16.435651 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.435649    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq podName:0971da9a-0556-4091-a7de-253ec8dce3e4 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:16.935632379 +0000 UTC m=+3.131209394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fm8rq" (UniqueName: "kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq") pod "network-check-target-zzbqh" (UID: "0971da9a-0556-4091-a7de-253ec8dce3e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:16.436198 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.436176    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzc85\" (UniqueName: \"kubernetes.io/projected/641d4e40-f66b-40c2-8db4-66b16d6958ad-kube-api-access-nzc85\") pod \"aws-ebs-csi-driver-node-z7ht5\" (UID: \"641d4e40-f66b-40c2-8db4-66b16d6958ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.438733 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.438712    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzv62\" (UniqueName: \"kubernetes.io/projected/c9602c0b-14d5-4878-9bb3-1b8914003013-kube-api-access-hzv62\") pod \"iptables-alerter-bb9dr\" (UID: \"c9602c0b-14d5-4878-9bb3-1b8914003013\") " pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.438825 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.438714    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n498b\" (UniqueName: \"kubernetes.io/projected/d622f883-4224-4801-b1d3-2b543c30dc35-kube-api-access-n498b\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:16.440123 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.440076    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal" event={"ID":"bf25b0953b6a4731ed485a31472da6c0","Type":"ContainerStarted","Data":"afc961d78ea154a5eb67adcc05ad3a0e134a069b30f110838d9431566d92b5df"}
Jul 07 11:40:16.441595 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.441570    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal" event={"ID":"46f616d7126e59bf8c9ddddc795b4398","Type":"ContainerStarted","Data":"2df63fc8dff63719bb0750e427556548771019d766227d07df5fe41257145bae"}
Jul 07 11:40:16.490283 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.490261    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fpndh"
Jul 07 11:40:16.499637 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.499618    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:16.504504 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:16.504478    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabea6595_66e7_4e5a_8414_e656fbe0b242.slice/crio-df348c21b02f8007fb875fb51165b294311815cbfaa5ebc34cfcf961e5388f92 WatchSource:0}: Error finding container df348c21b02f8007fb875fb51165b294311815cbfaa5ebc34cfcf961e5388f92: Status 404 returned error can't find the container with id df348c21b02f8007fb875fb51165b294311815cbfaa5ebc34cfcf961e5388f92
Jul 07 11:40:16.509634 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.509563    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdn6h"
Jul 07 11:40:16.517155 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.517138    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2rkx"
Jul 07 11:40:16.520957 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:16.520937    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda24ad97_d836_4d7a_9db2_586d261e3349.slice/crio-ee3882a37d6733d6a9104724873a4deca40018588e0aef86106ae9f655ce75ea WatchSource:0}: Error finding container ee3882a37d6733d6a9104724873a4deca40018588e0aef86106ae9f655ce75ea: Status 404 returned error can't find the container with id ee3882a37d6733d6a9104724873a4deca40018588e0aef86106ae9f655ce75ea
Jul 07 11:40:16.526747 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.526729    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zh7g2"
Jul 07 11:40:16.529449 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:16.529428    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9ceae5_ab86_4d83_b0ff_c5912fee77cd.slice/crio-27d200d2fcb3c7c0f3cf3b59b83fa8dd0f9e1e5d348139c1b09728e84786f79a WatchSource:0}: Error finding container 27d200d2fcb3c7c0f3cf3b59b83fa8dd0f9e1e5d348139c1b09728e84786f79a: Status 404 returned error can't find the container with id 27d200d2fcb3c7c0f3cf3b59b83fa8dd0f9e1e5d348139c1b09728e84786f79a
Jul 07 11:40:16.535056 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.535005    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nm6fk"
Jul 07 11:40:16.538503 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:16.538473    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3e2930_9990_4025_a22b_b53af21851e7.slice/crio-6675720be4b6561875c2305b61ae1d113ade70ed332769eda56814349b095e36 WatchSource:0}: Error finding container 6675720be4b6561875c2305b61ae1d113ade70ed332769eda56814349b095e36: Status 404 returned error can't find the container with id 6675720be4b6561875c2305b61ae1d113ade70ed332769eda56814349b095e36
Jul 07 11:40:16.544115 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.544055    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:16.546930 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:16.546905    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71053fbc_7333_4971_9c42_0d84b4fb6a6b.slice/crio-be864e418a292535ad5817f7fe2bd02e3363e9f04ba5638413305b56572c0687 WatchSource:0}: Error finding container be864e418a292535ad5817f7fe2bd02e3363e9f04ba5638413305b56572c0687: Status 404 returned error can't find the container with id be864e418a292535ad5817f7fe2bd02e3363e9f04ba5638413305b56572c0687
Jul 07 11:40:16.552789 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.552702    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5"
Jul 07 11:40:16.556062 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:16.556045    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb35b90_7072_43a8_81de_f8db577cd975.slice/crio-07bb1178521dd439c51e86301cfc0c640c5c5b19fa360020d720feea39a5dc80 WatchSource:0}: Error finding container 07bb1178521dd439c51e86301cfc0c640c5c5b19fa360020d720feea39a5dc80: Status 404 returned error can't find the container with id 07bb1178521dd439c51e86301cfc0c640c5c5b19fa360020d720feea39a5dc80
Jul 07 11:40:16.561641 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.561620    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bb9dr"
Jul 07 11:40:16.563163 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:16.563140    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641d4e40_f66b_40c2_8db4_66b16d6958ad.slice/crio-eebd9fa2d22cfd23e7268b4498d478a8e4c359952f32aade49d43fc761bd60d0 WatchSource:0}: Error finding container eebd9fa2d22cfd23e7268b4498d478a8e4c359952f32aade49d43fc761bd60d0: Status 404 returned error can't find the container with id eebd9fa2d22cfd23e7268b4498d478a8e4c359952f32aade49d43fc761bd60d0
Jul 07 11:40:16.571781 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:40:16.571760    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9602c0b_14d5_4878_9bb3_1b8914003013.slice/crio-23704941d509af6c43f4210b7078968e7a236c1986bc513bafcc5be4cb2c0168 WatchSource:0}: Error finding container 23704941d509af6c43f4210b7078968e7a236c1986bc513bafcc5be4cb2c0168: Status 404 returned error can't find the container with id 23704941d509af6c43f4210b7078968e7a236c1986bc513bafcc5be4cb2c0168
Jul 07 11:40:16.821397 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.821328    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:16.821532 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.821452    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:16.821532 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.821503    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret podName:66a5788d-0db4-4335-ae38-7e916eca3b86 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:17.821488069 +0000 UTC m=+4.017065069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret") pod "global-pull-secret-syncer-wrrg9" (UID: "66a5788d-0db4-4335-ae38-7e916eca3b86") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:16.922761 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:16.922724    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:16.922925 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.922848    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:16.922925 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:16.922914    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs podName:d622f883-4224-4801-b1d3-2b543c30dc35 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:17.922893752 +0000 UTC m=+4.118470755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs") pod "network-metrics-daemon-nwrkz" (UID: "d622f883-4224-4801-b1d3-2b543c30dc35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:17.023789 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.023714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:17.023949 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.023872    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:17.023949 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.023893    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:17.023949 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.023911    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fm8rq for pod openshift-network-diagnostics/network-check-target-zzbqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:17.024102 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.023986    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq podName:0971da9a-0556-4091-a7de-253ec8dce3e4 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:18.023964195 +0000 UTC m=+4.219541200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fm8rq" (UniqueName: "kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq") pod "network-check-target-zzbqh" (UID: "0971da9a-0556-4091-a7de-253ec8dce3e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:17.256774 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.256649    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 11:35:16 +0000 UTC" deadline="2027-12-03 03:00:29.228674834 +0000 UTC"
Jul 07 11:40:17.256774 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.256720    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12327h20m11.971958632s"
Jul 07 11:40:17.433768 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.433254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:17.433768 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.433381    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:17.445259 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.445190    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bb9dr" event={"ID":"c9602c0b-14d5-4878-9bb3-1b8914003013","Type":"ContainerStarted","Data":"23704941d509af6c43f4210b7078968e7a236c1986bc513bafcc5be4cb2c0168"}
Jul 07 11:40:17.449319 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.449227    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5" event={"ID":"641d4e40-f66b-40c2-8db4-66b16d6958ad","Type":"ContainerStarted","Data":"eebd9fa2d22cfd23e7268b4498d478a8e4c359952f32aade49d43fc761bd60d0"}
Jul 07 11:40:17.451156 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.451083    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-465p6" event={"ID":"5bb35b90-7072-43a8-81de-f8db577cd975","Type":"ContainerStarted","Data":"07bb1178521dd439c51e86301cfc0c640c5c5b19fa360020d720feea39a5dc80"}
Jul 07 11:40:17.453223 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.453165    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerStarted","Data":"be864e418a292535ad5817f7fe2bd02e3363e9f04ba5638413305b56572c0687"}
Jul 07 11:40:17.456050 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.455949    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"1deb7e90135cff5325bcb59136046a3f6ef44f58639cc1d92785da34b1796d79"}
Jul 07 11:40:17.458410 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.458383    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zh7g2" event={"ID":"cf3e2930-9990-4025-a22b-b53af21851e7","Type":"ContainerStarted","Data":"6675720be4b6561875c2305b61ae1d113ade70ed332769eda56814349b095e36"}
Jul 07 11:40:17.463590 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.463563    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2rkx" event={"ID":"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd","Type":"ContainerStarted","Data":"27d200d2fcb3c7c0f3cf3b59b83fa8dd0f9e1e5d348139c1b09728e84786f79a"}
Jul 07 11:40:17.465353 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.465327    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdn6h" event={"ID":"da24ad97-d836-4d7a-9db2-586d261e3349","Type":"ContainerStarted","Data":"ee3882a37d6733d6a9104724873a4deca40018588e0aef86106ae9f655ce75ea"}
Jul 07 11:40:17.467424 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.467370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fpndh" event={"ID":"abea6595-66e7-4e5a-8414-e656fbe0b242","Type":"ContainerStarted","Data":"df348c21b02f8007fb875fb51165b294311815cbfaa5ebc34cfcf961e5388f92"}
Jul 07 11:40:17.830739 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.830147    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:17.830739 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.830337    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:17.830739 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.830394    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret podName:66a5788d-0db4-4335-ae38-7e916eca3b86 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:19.830376832 +0000 UTC m=+6.025953849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret") pod "global-pull-secret-syncer-wrrg9" (UID: "66a5788d-0db4-4335-ae38-7e916eca3b86") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:17.931161 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:17.931125    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:17.931332 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.931317    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:17.931398 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:17.931382    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs podName:d622f883-4224-4801-b1d3-2b543c30dc35 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:19.931362344 +0000 UTC m=+6.126939360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs") pod "network-metrics-daemon-nwrkz" (UID: "d622f883-4224-4801-b1d3-2b543c30dc35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:18.032024 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:18.031995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:18.032180 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:18.032165    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:18.032224 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:18.032187    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:18.032224 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:18.032201    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fm8rq for pod openshift-network-diagnostics/network-check-target-zzbqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:18.032291 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:18.032254    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq podName:0971da9a-0556-4091-a7de-253ec8dce3e4 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:20.032237038 +0000 UTC m=+6.227814053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fm8rq" (UniqueName: "kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq") pod "network-check-target-zzbqh" (UID: "0971da9a-0556-4091-a7de-253ec8dce3e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:18.257069 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:18.257035    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 11:35:16 +0000 UTC" deadline="2028-04-11 11:02:09.567445966 +0000 UTC"
Jul 07 11:40:18.257069 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:18.257071    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15455h21m51.310379415s"
Jul 07 11:40:18.436840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:18.436321    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:18.436840 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:18.436451    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:18.436840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:18.436762    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:18.437368 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:18.436883    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:18.479143 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:18.478977    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal" event={"ID":"bf25b0953b6a4731ed485a31472da6c0","Type":"ContainerStarted","Data":"bf93d300b0b59b39f4e1ee3da2f77fe57ae4ddb7271be8f53f8b63f6594c892a"}
Jul 07 11:40:19.433597 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:19.433099    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:19.433597 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:19.433234    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:19.498130 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:19.497176    2572 generic.go:356] "Generic (PLEG): container finished" podID="46f616d7126e59bf8c9ddddc795b4398" containerID="bbcd43e14cac1890866f0bf9df687df05faff058c8b7a4b4a187ea985806e304" exitCode=0
Jul 07 11:40:19.498130 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:19.497846    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal" event={"ID":"46f616d7126e59bf8c9ddddc795b4398","Type":"ContainerDied","Data":"bbcd43e14cac1890866f0bf9df687df05faff058c8b7a4b4a187ea985806e304"}
Jul 07 11:40:19.514824 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:19.514748    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-177.ec2.internal" podStartSLOduration=4.514721014 podStartE2EDuration="4.514721014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:18.494379999 +0000 UTC m=+4.689957022" watchObservedRunningTime="2026-07-07 11:40:19.514721014 +0000 UTC m=+5.710298038"
Jul 07 11:40:19.846826 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:19.846789    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:19.847021 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:19.846943    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:19.847021 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:19.847006    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret podName:66a5788d-0db4-4335-ae38-7e916eca3b86 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:23.846987294 +0000 UTC m=+10.042564312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret") pod "global-pull-secret-syncer-wrrg9" (UID: "66a5788d-0db4-4335-ae38-7e916eca3b86") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:19.947273 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:19.947234    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:19.947434 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:19.947407    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:19.947491 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:19.947466    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs podName:d622f883-4224-4801-b1d3-2b543c30dc35 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:23.947447235 +0000 UTC m=+10.143024252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs") pod "network-metrics-daemon-nwrkz" (UID: "d622f883-4224-4801-b1d3-2b543c30dc35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:20.048579 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:20.048540    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:20.048790 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:20.048773    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:20.048856 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:20.048800    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:20.048856 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:20.048813    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fm8rq for pod openshift-network-diagnostics/network-check-target-zzbqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:20.048973 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:20.048866    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq podName:0971da9a-0556-4091-a7de-253ec8dce3e4 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:24.048848197 +0000 UTC m=+10.244425200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fm8rq" (UniqueName: "kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq") pod "network-check-target-zzbqh" (UID: "0971da9a-0556-4091-a7de-253ec8dce3e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:20.434250 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:20.434186    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:20.434415 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:20.434328    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:20.434496 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:20.434430    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:20.434569 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:20.434546    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:21.433596 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:21.433566    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:21.434030 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:21.433712    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:22.433763 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:22.433727    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:22.433763 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:22.433756    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:22.434241 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:22.433861    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:22.434241 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:22.433959    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:23.433546 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:23.433350    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:23.433546 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:23.433470    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:23.878003 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:23.877959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:23.878173 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:23.878107    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:23.878173 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:23.878167    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret podName:66a5788d-0db4-4335-ae38-7e916eca3b86 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:31.878151632 +0000 UTC m=+18.073728638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret") pod "global-pull-secret-syncer-wrrg9" (UID: "66a5788d-0db4-4335-ae38-7e916eca3b86") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:23.978870 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:23.978832    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:23.979052 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:23.979009    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:23.979109 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:23.979091    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs podName:d622f883-4224-4801-b1d3-2b543c30dc35 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:31.979070022 +0000 UTC m=+18.174647034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs") pod "network-metrics-daemon-nwrkz" (UID: "d622f883-4224-4801-b1d3-2b543c30dc35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:24.080344 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:24.080211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:24.080528 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:24.080385    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:24.080528 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:24.080404    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:24.080528 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:24.080418    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fm8rq for pod openshift-network-diagnostics/network-check-target-zzbqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:24.080528 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:24.080485    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq podName:0971da9a-0556-4091-a7de-253ec8dce3e4 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:32.080465264 +0000 UTC m=+18.276042282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fm8rq" (UniqueName: "kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq") pod "network-check-target-zzbqh" (UID: "0971da9a-0556-4091-a7de-253ec8dce3e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:24.434441 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:24.434407    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:24.434871 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:24.434542    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:24.434871 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:24.434585    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:24.434871 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:24.434706    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:25.433534 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:25.433496    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:25.433761 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:25.433623    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:26.433563 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:26.433518    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:26.434024 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:26.433706    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:26.434024 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:26.433544    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:26.434142 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:26.434105    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:27.433494 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:27.433463    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:27.433652 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:27.433575    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:28.433161 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:28.433132    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:28.433326 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:28.433132    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:28.433326 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:28.433254    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:28.433447 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:28.433359    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:29.433161 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:29.433126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:29.433547 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:29.433253    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:30.433724 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:30.433690    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:30.434176 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:30.433823    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:30.434176 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:30.433895    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:30.434176 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:30.434004    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:31.433918 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:31.433843    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:31.434322 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:31.433955    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:31.938777 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:31.938745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:31.938950 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:31.938869    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:31.938950 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:31.938933    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret podName:66a5788d-0db4-4335-ae38-7e916eca3b86 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:47.938911639 +0000 UTC m=+34.134488654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret") pod "global-pull-secret-syncer-wrrg9" (UID: "66a5788d-0db4-4335-ae38-7e916eca3b86") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:32.039186 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:32.039152    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:32.039335 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:32.039302    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:32.039379 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:32.039370    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs podName:d622f883-4224-4801-b1d3-2b543c30dc35 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:48.039352907 +0000 UTC m=+34.234929914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs") pod "network-metrics-daemon-nwrkz" (UID: "d622f883-4224-4801-b1d3-2b543c30dc35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:32.140404 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:32.140371    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:32.140535 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:32.140506    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:32.140535 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:32.140524    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:32.140535 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:32.140535    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fm8rq for pod openshift-network-diagnostics/network-check-target-zzbqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:32.140644 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:32.140578    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq podName:0971da9a-0556-4091-a7de-253ec8dce3e4 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:48.140564957 +0000 UTC m=+34.336141962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fm8rq" (UniqueName: "kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq") pod "network-check-target-zzbqh" (UID: "0971da9a-0556-4091-a7de-253ec8dce3e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:32.433781 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:32.433745    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:32.433962 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:32.433883    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:32.434310 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:32.433746    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:32.434363 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:32.434336    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:33.433995 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:33.433964    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:33.434435 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:33.434090    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:34.434750 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:34.434718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:34.435193 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:34.434831    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:34.435193 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:34.434887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:34.435193 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:34.434981    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:35.433789 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.433511    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:35.433789 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:35.433760    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:35.528090 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.528057    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2rkx" event={"ID":"5d9ceae5-ab86-4d83-b0ff-c5912fee77cd","Type":"ContainerStarted","Data":"c72ad7df5e13e9945470c5598dbc8111214bf1ab182562e923fd49c647923865"}
Jul 07 11:40:35.529579 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.529553    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdn6h" event={"ID":"da24ad97-d836-4d7a-9db2-586d261e3349","Type":"ContainerStarted","Data":"5caeeecc126fd05c0016b5f73a59b82f26aaa5ff946bce5fab08ff557d5022d5"}
Jul 07 11:40:35.531006 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.530985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fpndh" event={"ID":"abea6595-66e7-4e5a-8414-e656fbe0b242","Type":"ContainerStarted","Data":"25793a11e4fbca85d9b8e3a013157cc6d0ff568c627a8b7719a348a51afd829e"}
Jul 07 11:40:35.532955 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.532930    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal" event={"ID":"46f616d7126e59bf8c9ddddc795b4398","Type":"ContainerStarted","Data":"bc6519c4c159e7065606a6eafeb5e8077ecf2a4816d752af55337bee84363431"}
Jul 07 11:40:35.534365 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.534341    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5" event={"ID":"641d4e40-f66b-40c2-8db4-66b16d6958ad","Type":"ContainerStarted","Data":"668f4c6c0d0ab911db98505523eaab1afb75a97d780af6be703c55fb3dfd6d08"}
Jul 07 11:40:35.535619 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.535594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-465p6" event={"ID":"5bb35b90-7072-43a8-81de-f8db577cd975","Type":"ContainerStarted","Data":"6a75c87f39798399c541a314538ca37d967767e686eb1bca1ee056e79a59a5cd"}
Jul 07 11:40:35.537213 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.537188    2572 generic.go:356] "Generic (PLEG): container finished" podID="71053fbc-7333-4971-9c42-0d84b4fb6a6b" containerID="c0c2c7c6431a2b7809f7c84530abc55bf69af7580f3a671a8e3519a18692d02a" exitCode=0
Jul 07 11:40:35.537329 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.537269    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerDied","Data":"c0c2c7c6431a2b7809f7c84530abc55bf69af7580f3a671a8e3519a18692d02a"}
Jul 07 11:40:35.539111 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.539090    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zh7g2" event={"ID":"cf3e2930-9990-4025-a22b-b53af21851e7","Type":"ContainerStarted","Data":"ffba0b9674384da5ad57a17877e2ba0aa28d0519993d3e95ad0f25f60dbbe76d"}
Jul 07 11:40:35.567452 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.567370    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g2rkx" podStartSLOduration=6.993443229 podStartE2EDuration="21.567356268s" totalImagesPullingTime="14.573913039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.544911661 +0000 UTC m=+21.740488693" watchObservedRunningTime="2026-07-07 11:40:35.567356268 +0000 UTC m=+21.762933357"
Jul 07 11:40:35.581811 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.581769    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-465p6" podStartSLOduration=2.952725088 podStartE2EDuration="21.581755654s" totalImagesPullingTime="18.629030566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.581182919 +0000 UTC m=+21.776759942" watchObservedRunningTime="2026-07-07 11:40:35.581755654 +0000 UTC m=+21.777332676"
Jul 07 11:40:35.600104 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.600067    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-177.ec2.internal" podStartSLOduration=20.600055512 podStartE2EDuration="20.600055512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.599672423 +0000 UTC m=+21.795249439" watchObservedRunningTime="2026-07-07 11:40:35.600055512 +0000 UTC m=+21.795632534"
Jul 07 11:40:35.619874 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.619841    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fpndh" podStartSLOduration=2.8960097620000003 podStartE2EDuration="21.619829435s" totalImagesPullingTime="18.723819673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.619464924 +0000 UTC m=+21.815041945" watchObservedRunningTime="2026-07-07 11:40:35.619829435 +0000 UTC m=+21.815406457"
Jul 07 11:40:35.659375 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.659341    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xdn6h" podStartSLOduration=2.997140919 podStartE2EDuration="21.659330738s" totalImagesPullingTime="18.662189819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.656471888 +0000 UTC m=+21.852048914" watchObservedRunningTime="2026-07-07 11:40:35.659330738 +0000 UTC m=+21.854907759"
Jul 07 11:40:35.689299 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.689262    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zh7g2" podStartSLOduration=3.042949191 podStartE2EDuration="21.68924775s" totalImagesPullingTime="18.646298559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.689046942 +0000 UTC m=+21.884623965" watchObservedRunningTime="2026-07-07 11:40:35.68924775 +0000 UTC m=+21.884824771"
Jul 07 11:40:35.692041 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.692014    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:35.777208 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.777067    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:35.882429 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.882126    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:35.979273 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:35.979244    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:36.369121 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.369007    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:36.399344 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.399323    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:36.400046 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.400024    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:36.433492 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.433447    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:36.433625 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:36.433567    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:36.433749 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.433682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:36.433957 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:36.433931    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:36.435277 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.435246    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:36.506950 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.506925    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:36.542902 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.542886    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:40:36.543276 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.543188    2572 generic.go:356] "Generic (PLEG): container finished" podID="cad78671-89ed-4a34-877e-7e0448976883" containerID="4387bb3390ff223c782c5d90cc7f7d96a2a237f229e087f272b626b877c0c554" exitCode=1
Jul 07 11:40:36.543329 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.543268    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"89569222d575259eb63c9660404689dafa131d7ed84a47805ba5bf9a175f89f1"}
Jul 07 11:40:36.543378 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.543339    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"659140774c2b43c9f9de628caf399d19954a7cb0d4705e1f98a334e90a13851c"}
Jul 07 11:40:36.543418 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.543385    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"a66566291706ea76bc68ff6bdd0cea498533f617c5df9f493e3b4d7bb291b76e"}
Jul 07 11:40:36.543459 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.543417    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerDied","Data":"4387bb3390ff223c782c5d90cc7f7d96a2a237f229e087f272b626b877c0c554"}
Jul 07 11:40:36.543459 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.543447    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"aa7f86f32f15b2034cd3bd50ca47af5a65c69893accfffbf210c6eafa0c6353b"}
Jul 07 11:40:36.646522 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.646444    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:36.748288 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.748262    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:36.832436 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.832404    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 11:40:36.925770 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.925713    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:36.926265 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:36.926247    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-465p6"
Jul 07 11:40:37.004995 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.004965    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:37.098588 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.098415    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:37.187677 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.187591    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:37.295353 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.295247    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T11:40:36.832433092Z","UUID":"2c485eef-2331-4a4b-a36c-bfcead5e9e82","Handler":null,"Name":"","Endpoint":""}
Jul 07 11:40:37.296926 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.296899    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 11:40:37.297096 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.296958    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 11:40:37.433636 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.433602    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:37.433880 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:37.433736    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:37.548549 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.548517    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5" event={"ID":"641d4e40-f66b-40c2-8db4-66b16d6958ad","Type":"ContainerStarted","Data":"d78fcbf5f9ff3124bd43aeaccd9f793812bf224297812e2c48c3a8a6dfe32d87"}
Jul 07 11:40:37.551795 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.551772    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:40:37.552321 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.552290    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"122944837f851507962bd6989b990a61e8221ce37728ba7cbdf8e893671c000c"}
Jul 07 11:40:37.553988 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.553926    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bb9dr" event={"ID":"c9602c0b-14d5-4878-9bb3-1b8914003013","Type":"ContainerStarted","Data":"336184e3d309b6e2eca7bac34f4d374787f2cb7ea8846989e285897a1d6390d5"}
Jul 07 11:40:37.582944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.582896    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bb9dr" podStartSLOduration=4.96938688 podStartE2EDuration="23.582878978s" totalImagesPullingTime="18.613492098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:37.582627987 +0000 UTC m=+23.778205009" watchObservedRunningTime="2026-07-07 11:40:37.582878978 +0000 UTC m=+23.778456002"
Jul 07 11:40:37.887250 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:37.887172    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:38.254360 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:38.254327    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:38.320578 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:38.320547    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:38.436756 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:38.436729    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:38.436895 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:38.436845    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:38.436895 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:38.436874    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:38.437007 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:38.436930    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:38.558391 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:38.558315    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5" event={"ID":"641d4e40-f66b-40c2-8db4-66b16d6958ad","Type":"ContainerStarted","Data":"a3e7b6255063b08483cf650d6df20b9ec3cb494979abb638d3db8e3775db1815"}
Jul 07 11:40:38.578947 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:38.578901    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7ht5" podStartSLOduration=3.577158111 podStartE2EDuration="24.578888344s" totalImagesPullingTime="21.001730233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:38.578604836 +0000 UTC m=+24.774181856" watchObservedRunningTime="2026-07-07 11:40:38.578888344 +0000 UTC m=+24.774465376"
Jul 07 11:40:39.433978 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:39.433948    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:39.434169 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:39.434071    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:40.281336 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.281306    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:40.433083 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.433034    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:40.433177 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.433034    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:40.433177 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:40.433148    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:40.433241 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:40.433222    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:40.489133 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.489103    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:40.563940 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.563919    2572 generic.go:356] "Generic (PLEG): container finished" podID="71053fbc-7333-4971-9c42-0d84b4fb6a6b" containerID="7f4864d1972ab13efe5668b8b37d0bce2f8d5cb7d659270ea556610c77c5cf7a" exitCode=0
Jul 07 11:40:40.564050 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.563984    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerDied","Data":"7f4864d1972ab13efe5668b8b37d0bce2f8d5cb7d659270ea556610c77c5cf7a"}
Jul 07 11:40:40.565591 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.565560    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:40.567317 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.567299    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:40:40.567711 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.567693    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"e422afce680f42ab814fd4984f401f2046c94636fe27c8a80ab099593a1f9b30"}
Jul 07 11:40:40.775459 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.775433    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:40.884458 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:40.884438    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:41.053376 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:41.053326    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:41.154118 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:41.154093    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:41.434072 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:41.433996    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:41.434431 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:41.434097    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:41.534110 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:41.534083    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:41.571522 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:41.571499    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerStarted","Data":"aac854b764568d9a14caf8e978a62a7b9aec12b1dfa74905d77f4c4612539497"}
Jul 07 11:40:41.656721 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:41.656683    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:41.853301 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:41.853282    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2rkx_5d9ceae5-ab86-4d83-b0ff-c5912fee77cd/dns-node-resolver/0.log"
Jul 07 11:40:42.436477 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.436274    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:42.436877 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.436329    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:42.436877 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:42.436508    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:42.436877 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:42.436570    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:42.575483 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.575452    2572 generic.go:356] "Generic (PLEG): container finished" podID="71053fbc-7333-4971-9c42-0d84b4fb6a6b" containerID="aac854b764568d9a14caf8e978a62a7b9aec12b1dfa74905d77f4c4612539497" exitCode=0
Jul 07 11:40:42.575593 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.575532    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerDied","Data":"aac854b764568d9a14caf8e978a62a7b9aec12b1dfa74905d77f4c4612539497"}
Jul 07 11:40:42.579242 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.579227    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:40:42.579680 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.579644    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"80a4fb85e5d4c7e14788f774bbd5e514f812331c06973d203d653ea210dad0dc"}
Jul 07 11:40:42.579972 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.579954    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:42.580073 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.580015    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:42.580129 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.580101    2572 scope.go:117] "RemoveContainer" containerID="4387bb3390ff223c782c5d90cc7f7d96a2a237f229e087f272b626b877c0c554"
Jul 07 11:40:42.581245 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.581162    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:42.596247 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.596229    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:42.664466 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.664447    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zh7g2_cf3e2930-9990-4025-a22b-b53af21851e7/node-ca/0.log"
Jul 07 11:40:42.692231 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:42.692170    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:43.433440 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.433405    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:43.433593 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:43.433525    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:43.586911 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.586892    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:40:43.587810 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.587321    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" event={"ID":"cad78671-89ed-4a34-877e-7e0448976883","Type":"ContainerStarted","Data":"d8c8e10cdcc212910cb3d61c1c2f4911e16a941120f2cac86a188564b5d81c4e"}
Jul 07 11:40:43.587810 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.587603    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:43.609189 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.609157    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:40:43.640646 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.640606    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv" podStartSLOduration=10.503789843 podStartE2EDuration="29.640589632s" totalImagesPullingTime="19.136799789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:43.639225014 +0000 UTC m=+29.834802037" watchObservedRunningTime="2026-07-07 11:40:43.640589632 +0000 UTC m=+29.836166660"
Jul 07 11:40:43.823335 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.823304    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:43.952916 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.952688    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zzbqh"]
Jul 07 11:40:43.953051 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.953029    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:43.953159 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:43.953136    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:43.957459 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.957438    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wrrg9"]
Jul 07 11:40:43.957575 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.957525    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:43.957652 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:43.957600    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:43.967930 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.967910    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwrkz"]
Jul 07 11:40:43.968022 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:43.967998    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:43.968113 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:43.968093    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:44.463047 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:44.462977    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:44.591938 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:44.591905    2572 generic.go:356] "Generic (PLEG): container finished" podID="71053fbc-7333-4971-9c42-0d84b4fb6a6b" containerID="883ec97f026901417fe136fd515d34b69e4461d9867961ef9b99607ea3f239ec" exitCode=0
Jul 07 11:40:44.592395 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:44.591985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerDied","Data":"883ec97f026901417fe136fd515d34b69e4461d9867961ef9b99607ea3f239ec"}
Jul 07 11:40:44.719001 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:44.718929    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:44.830417 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:44.830384    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:45.079699 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:45.079653    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:45.276220 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:45.276193    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:45.434132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:45.434056    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:45.434132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:45.434070    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:45.434132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:45.434070    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:45.434378 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:45.434174    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:45.434378 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:45.434240    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:45.434378 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:45.434337    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:46.329720 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:46.329684    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:46.533823 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:46.533789    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:46.753044 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:46.753010    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:46.920897 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:46.920855    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:47.164718 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:47.164542    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:47.298619 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:47.298561    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:47.434169 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:47.434104    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:47.434169 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:47.434126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:47.434781 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:47.434114    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:47.434781 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:47.434224    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:47.434781 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:47.434288    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:47.434781 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:47.434388    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:47.801866 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:47.801828    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:47.899438 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:47.899412    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:47.969371 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:47.969345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:47.969491 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:47.969452    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:47.969527 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:47.969503    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret podName:66a5788d-0db4-4335-ae38-7e916eca3b86 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:19.969487914 +0000 UTC m=+66.165064932 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret") pod "global-pull-secret-syncer-wrrg9" (UID: "66a5788d-0db4-4335-ae38-7e916eca3b86") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:48.070448 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.070391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:48.070538 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:48.070500    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:48.070572 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:48.070547    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs podName:d622f883-4224-4801-b1d3-2b543c30dc35 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:20.070533125 +0000 UTC m=+66.266110143 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs") pod "network-metrics-daemon-nwrkz" (UID: "d622f883-4224-4801-b1d3-2b543c30dc35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:48.099985 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.099959    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:48.170966 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.170944    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:48.171074 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:48.171063    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:48.171118 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:48.171080    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:48.171118 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:48.171089    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fm8rq for pod openshift-network-diagnostics/network-check-target-zzbqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:48.171176 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:48.171125    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq podName:0971da9a-0556-4091-a7de-253ec8dce3e4 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:20.171115591 +0000 UTC m=+66.366692592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fm8rq" (UniqueName: "kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq") pod "network-check-target-zzbqh" (UID: "0971da9a-0556-4091-a7de-253ec8dce3e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:48.202271 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.202248    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:48.331417 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.331344    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:48.501393 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.501357    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:48.715917 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.715842    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:48.826701 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.826653    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:48.916153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:48.916123    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:49.090288 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:49.090233    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:49.372152 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:49.372064    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:49.433183 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:49.433150    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:49.433183 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:49.433178    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:49.433396 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:49.433150    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:49.433396 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:49.433277    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:49.433396 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:49.433366    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:49.433546 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:49.433442    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:50.102965 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:50.102929    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:50.397758 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:50.397721    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:50.496125 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:50.496042    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:50.598146 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:50.598112    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:50.606856 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:50.606829    2572 generic.go:356] "Generic (PLEG): container finished" podID="71053fbc-7333-4971-9c42-0d84b4fb6a6b" containerID="70f94b4eee0cae05d99fdd39fd649c76d698f1bd6a7c74f68831b72c093f4a1a" exitCode=0
Jul 07 11:40:50.606948 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:50.606883    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerDied","Data":"70f94b4eee0cae05d99fdd39fd649c76d698f1bd6a7c74f68831b72c093f4a1a"}
Jul 07 11:40:50.779906 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:50.779879    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:50.925310 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:50.925252    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:51.091104 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:51.091082    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:51.266182 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:51.266154    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:51.433074 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:51.433050    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:51.433214 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:51.433134    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:51.433214 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:51.433144    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:51.433298 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:51.433212    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:51.433298 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:51.433222    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:51.433358 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:51.433302    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:51.612294 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:51.612220    2572 generic.go:356] "Generic (PLEG): container finished" podID="71053fbc-7333-4971-9c42-0d84b4fb6a6b" containerID="202f96c7ce5251a814f77eeee1d8ff85013f6174cc070f5bcc3c1498bbf82cc5" exitCode=0
Jul 07 11:40:51.612294 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:51.612278    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerDied","Data":"202f96c7ce5251a814f77eeee1d8ff85013f6174cc070f5bcc3c1498bbf82cc5"}
Jul 07 11:40:52.153062 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:52.152867    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:52.503981 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:52.503945    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:52.617026 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:52.616996    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" event={"ID":"71053fbc-7333-4971-9c42-0d84b4fb6a6b","Type":"ContainerStarted","Data":"3bbb8bd37558a24d73367a13c06320c414e511ba6efdc7361cbc82b1813b94de"}
Jul 07 11:40:52.874569 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:52.874505    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:52.971822 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:52.971792    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:53.116044 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:53.116020    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:53.433616 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:53.433595    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:53.433616 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:53.433608    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:53.433762 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:53.433704    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:53.433762 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:53.433730    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:53.433831 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:53.433791    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:53.433865 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:53.433855    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:53.691274 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:53.691204    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:53.791451 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:53.791423    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:54.503548 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:54.503515    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:54.696416 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:54.696384    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:54.890961 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:54.890896    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:55.091029 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:55.090994    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:40:55.434091 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:55.434064    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:55.434091 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:55.434087    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:55.434294 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:55.434074    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:55.434294 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:55.434163    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:55.434294 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:55.434257    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:55.434385 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:55.434309    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:57.433522 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:57.433487    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:57.433522 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:57.433517    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:57.433941 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:57.433577    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:57.433941 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:57.433620    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:57.433941 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:57.433724    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:40:57.433941 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:57.433757    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:59.433820 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:59.433792    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:40:59.434212 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:59.433795    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:40:59.434212 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:40:59.433796    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:40:59.434212 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:59.433961    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:40:59.434212 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:59.433880    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:40:59.434212 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:40:59.434038    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:41:00.103357 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:00.103328    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:01.433186 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:01.433158    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:41:01.433539 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:01.433188    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:01.433539 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:01.433248    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:41:01.433539 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:01.433278    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:41:01.433539 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:01.433354    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:41:01.433539 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:01.433412    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:41:03.433642 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:03.433613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:41:03.433642 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:03.433643    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:41:03.434170 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:03.433735    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:41:03.434170 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:03.433805    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:41:03.434170 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:03.433845    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:03.434170 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:03.433915    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:41:05.401131 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:05.401089    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:05.433573 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:05.433550    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:41:05.433704 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:05.433551    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:41:05.433704 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:05.433650    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:41:05.433789 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:05.433551    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:05.433789 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:05.433739    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:41:05.433789 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:05.433779    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:41:05.554887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:05.554855    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:07.401031 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:07.400970    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nm6fk" podStartSLOduration=26.70106417 podStartE2EDuration="53.400951033s" totalImagesPullingTime="26.699886863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:52.659837781 +0000 UTC m=+38.855414804" watchObservedRunningTime="2026-07-07 11:41:07.400951033 +0000 UTC m=+53.596528058"
Jul 07 11:41:07.433127 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:07.433093    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:41:07.433265 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:07.433202    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:41:07.433265 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:07.433212    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrrg9" podUID="66a5788d-0db4-4335-ae38-7e916eca3b86"
Jul 07 11:41:07.433449 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:07.433305    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwrkz" podUID="d622f883-4224-4801-b1d3-2b543c30dc35"
Jul 07 11:41:07.433449 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:07.433212    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:07.433449 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:07.433384    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzbqh" podUID="0971da9a-0556-4091-a7de-253ec8dce3e4"
Jul 07 11:41:07.718632 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:07.718560    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:08.014621 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.014589    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:08.060221 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.060192    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-177.ec2.internal" event="NodeReady"
Jul 07 11:41:08.060387 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.060336    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 11:41:08.379332 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.379264    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bxcm5"]
Jul 07 11:41:08.380769 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.380753    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bxcm5"
Jul 07 11:41:08.383902 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.383883    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xzh75\""
Jul 07 11:41:08.384004 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.383906    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 11:41:08.385202 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.385188    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 11:41:08.385357 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.385339    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 11:41:08.422709 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.422688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzcv\" (UniqueName: \"kubernetes.io/projected/dd7c56e7-36b1-41e8-9624-2ca61840b3f3-kube-api-access-2vzcv\") pod \"ingress-canary-bxcm5\" (UID: \"dd7c56e7-36b1-41e8-9624-2ca61840b3f3\") " pod="openshift-ingress-canary/ingress-canary-bxcm5"
Jul 07 11:41:08.422978 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.422723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd7c56e7-36b1-41e8-9624-2ca61840b3f3-cert\") pod \"ingress-canary-bxcm5\" (UID: \"dd7c56e7-36b1-41e8-9624-2ca61840b3f3\") " pod="openshift-ingress-canary/ingress-canary-bxcm5"
Jul 07 11:41:08.441654 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.441635    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2tjsn"]
Jul 07 11:41:08.443439 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.443424    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.446813 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.446797    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 11:41:08.446893 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.446875    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 11:41:08.446946 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.446877    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 11:41:08.447005 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.446986    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 11:41:08.447781 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.447754    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t6ln8\""
Jul 07 11:41:08.452341 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.452321    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:08.465155 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.465132    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7r6ns"]
Jul 07 11:41:08.466791 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.466776    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2tjsn"]
Jul 07 11:41:08.466853 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.466848    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.502773 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.502755    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pn956\""
Jul 07 11:41:08.502868 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.502859    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 11:41:08.503057 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.503042    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 11:41:08.503442 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.503424    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bxcm5"]
Jul 07 11:41:08.516539 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.516521    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7r6ns"]
Jul 07 11:41:08.523201 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523181    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/87c9f985-a645-4f3a-a284-1f515c5cb2a2-config-volume\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.523287 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523206    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b954c61a-ecae-42d2-9fd3-0c931f5cda90-crio-socket\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.523287 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523234    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd7c56e7-36b1-41e8-9624-2ca61840b3f3-cert\") pod \"ingress-canary-bxcm5\" (UID: \"dd7c56e7-36b1-41e8-9624-2ca61840b3f3\") " pod="openshift-ingress-canary/ingress-canary-bxcm5"
Jul 07 11:41:08.523287 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523276    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b954c61a-ecae-42d2-9fd3-0c931f5cda90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.523390 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523348    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8q9z\" (UniqueName: \"kubernetes.io/projected/b954c61a-ecae-42d2-9fd3-0c931f5cda90-kube-api-access-b8q9z\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.523430 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b954c61a-ecae-42d2-9fd3-0c931f5cda90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.523467 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523431    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87c9f985-a645-4f3a-a284-1f515c5cb2a2-tmp-dir\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.523467 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523446    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b954c61a-ecae-42d2-9fd3-0c931f5cda90-data-volume\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.523533 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523479    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87c9f985-a645-4f3a-a284-1f515c5cb2a2-metrics-tls\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.523533 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523506    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw2gx\" (UniqueName: \"kubernetes.io/projected/87c9f985-a645-4f3a-a284-1f515c5cb2a2-kube-api-access-zw2gx\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.523604 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.523538    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzcv\" (UniqueName: \"kubernetes.io/projected/dd7c56e7-36b1-41e8-9624-2ca61840b3f3-kube-api-access-2vzcv\") pod \"ingress-canary-bxcm5\" (UID: \"dd7c56e7-36b1-41e8-9624-2ca61840b3f3\") " pod="openshift-ingress-canary/ingress-canary-bxcm5"
Jul 07 11:41:08.527253 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.527234    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd7c56e7-36b1-41e8-9624-2ca61840b3f3-cert\") pod \"ingress-canary-bxcm5\" (UID: \"dd7c56e7-36b1-41e8-9624-2ca61840b3f3\") " pod="openshift-ingress-canary/ingress-canary-bxcm5"
Jul 07 11:41:08.552041 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.552014    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzcv\" (UniqueName: \"kubernetes.io/projected/dd7c56e7-36b1-41e8-9624-2ca61840b3f3-kube-api-access-2vzcv\") pod \"ingress-canary-bxcm5\" (UID: \"dd7c56e7-36b1-41e8-9624-2ca61840b3f3\") " pod="openshift-ingress-canary/ingress-canary-bxcm5"
Jul 07 11:41:08.624462 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624430    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b954c61a-ecae-42d2-9fd3-0c931f5cda90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.624602 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624505    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87c9f985-a645-4f3a-a284-1f515c5cb2a2-tmp-dir\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.624602 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624533    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b954c61a-ecae-42d2-9fd3-0c931f5cda90-data-volume\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.624602 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624560    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87c9f985-a645-4f3a-a284-1f515c5cb2a2-metrics-tls\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.624602 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624582    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2gx\" (UniqueName: \"kubernetes.io/projected/87c9f985-a645-4f3a-a284-1f515c5cb2a2-kube-api-access-zw2gx\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.624843 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624717    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/87c9f985-a645-4f3a-a284-1f515c5cb2a2-config-volume\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.624843 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624756    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b954c61a-ecae-42d2-9fd3-0c931f5cda90-crio-socket\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.624843 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b954c61a-ecae-42d2-9fd3-0c931f5cda90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.624843 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624831    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8q9z\" (UniqueName: \"kubernetes.io/projected/b954c61a-ecae-42d2-9fd3-0c931f5cda90-kube-api-access-b8q9z\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.625030 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624961    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b954c61a-ecae-42d2-9fd3-0c931f5cda90-data-volume\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.625030 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.624967    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87c9f985-a645-4f3a-a284-1f515c5cb2a2-tmp-dir\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.625142 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.625036    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b954c61a-ecae-42d2-9fd3-0c931f5cda90-crio-socket\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.625388 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.625366    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b954c61a-ecae-42d2-9fd3-0c931f5cda90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.625455 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.625368    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/87c9f985-a645-4f3a-a284-1f515c5cb2a2-config-volume\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.626863 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.626842    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b954c61a-ecae-42d2-9fd3-0c931f5cda90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.626962 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.626933    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87c9f985-a645-4f3a-a284-1f515c5cb2a2-metrics-tls\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.671729 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.671646    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:08.672200 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.672167    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2gx\" (UniqueName: \"kubernetes.io/projected/87c9f985-a645-4f3a-a284-1f515c5cb2a2-kube-api-access-zw2gx\") pod \"dns-default-7r6ns\" (UID: \"87c9f985-a645-4f3a-a284-1f515c5cb2a2\") " pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.672913 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.672892    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8q9z\" (UniqueName: \"kubernetes.io/projected/b954c61a-ecae-42d2-9fd3-0c931f5cda90-kube-api-access-b8q9z\") pod \"insights-runtime-extractor-2tjsn\" (UID: \"b954c61a-ecae-42d2-9fd3-0c931f5cda90\") " pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.689797 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.689776    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bxcm5"
Jul 07 11:41:08.752573 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.752319    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2tjsn"
Jul 07 11:41:08.774257 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.773974    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:08.842467 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.842439    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bxcm5"]
Jul 07 11:41:08.850169 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:08.850109    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7c56e7_36b1_41e8_9624_2ca61840b3f3.slice/crio-16b0a3a4dd59601002966842ddadc1094487cc093b99fd79abe6c8fdef7d7e0e WatchSource:0}: Error finding container 16b0a3a4dd59601002966842ddadc1094487cc093b99fd79abe6c8fdef7d7e0e: Status 404 returned error can't find the container with id 16b0a3a4dd59601002966842ddadc1094487cc093b99fd79abe6c8fdef7d7e0e
Jul 07 11:41:08.852975 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.852749    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:08.927989 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.927925    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2tjsn"]
Jul 07 11:41:08.931336 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:08.931313    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb954c61a_ecae_42d2_9fd3_0c931f5cda90.slice/crio-d57e17b24efc69825a6769c1d7663fbebb8e08645a917aa3f47fd332163df077 WatchSource:0}: Error finding container d57e17b24efc69825a6769c1d7663fbebb8e08645a917aa3f47fd332163df077: Status 404 returned error can't find the container with id d57e17b24efc69825a6769c1d7663fbebb8e08645a917aa3f47fd332163df077
Jul 07 11:41:08.976726 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:08.976706    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7r6ns"]
Jul 07 11:41:08.978553 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:08.978530    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c9f985_a645_4f3a_a284_1f515c5cb2a2.slice/crio-4eafa1aa8246e407c49c620b1c2968ef45a7c729e04029935d366aa56423a09a WatchSource:0}: Error finding container 4eafa1aa8246e407c49c620b1c2968ef45a7c729e04029935d366aa56423a09a: Status 404 returned error can't find the container with id 4eafa1aa8246e407c49c620b1c2968ef45a7c729e04029935d366aa56423a09a
Jul 07 11:41:09.055288 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.055248    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:09.433789 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.433757    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:41:09.434255 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.433764    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:41:09.434255 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.433764    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:09.439062 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.439040    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 11:41:09.440182 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.439626    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 11:41:09.440182 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.439725    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5vsqg\""
Jul 07 11:41:09.440182 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.439851    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z7ksr\""
Jul 07 11:41:09.440182 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.439904    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 11:41:09.440182 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.440037    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 11:41:09.654700 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.654648    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bxcm5" event={"ID":"dd7c56e7-36b1-41e8-9624-2ca61840b3f3","Type":"ContainerStarted","Data":"16b0a3a4dd59601002966842ddadc1094487cc093b99fd79abe6c8fdef7d7e0e"}
Jul 07 11:41:09.657621 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.657590    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r6ns" event={"ID":"87c9f985-a645-4f3a-a284-1f515c5cb2a2","Type":"ContainerStarted","Data":"4eafa1aa8246e407c49c620b1c2968ef45a7c729e04029935d366aa56423a09a"}
Jul 07 11:41:09.661433 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.661402    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2tjsn" event={"ID":"b954c61a-ecae-42d2-9fd3-0c931f5cda90","Type":"ContainerStarted","Data":"d57e17b24efc69825a6769c1d7663fbebb8e08645a917aa3f47fd332163df077"}
Jul 07 11:41:09.674889 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.674864    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:09.784144 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:09.784068    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:12.448893 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:12.448850    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:12.653002 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:12.652968    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:12.790086 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:12.790054    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:12.941291 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:12.941258    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:13.051817 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:13.051747    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:13.893997 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:13.893960    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:14.032920 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:14.032884    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:14.678718 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:14.678682    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bxcm5" event={"ID":"dd7c56e7-36b1-41e8-9624-2ca61840b3f3","Type":"ContainerStarted","Data":"eb2f1d65a54d64daec80c8d49ee45b947dccdd181bad75bfa50cf6904123c543"}
Jul 07 11:41:14.680933 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:14.680902    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r6ns" event={"ID":"87c9f985-a645-4f3a-a284-1f515c5cb2a2","Type":"ContainerStarted","Data":"9ce04659936fa5ea8d91baf917b2b2a29afd04922ead597a6dcf9a09e286166c"}
Jul 07 11:41:14.681079 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:14.680956    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r6ns" event={"ID":"87c9f985-a645-4f3a-a284-1f515c5cb2a2","Type":"ContainerStarted","Data":"ea53aeb86435c796372581a3aa5898e48b5076c08f96dc8fd7e9f8536c946563"}
Jul 07 11:41:14.681079 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:14.680985    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:14.682624 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:14.682597    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2tjsn" event={"ID":"b954c61a-ecae-42d2-9fd3-0c931f5cda90","Type":"ContainerStarted","Data":"43c09263db07e93768228d3498e472050fa57803ebc3ec3c39c8dcb6812185b6"}
Jul 07 11:41:15.080163 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.080135    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:15.173256 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.173212    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bxcm5" podStartSLOduration=1.835265172 podStartE2EDuration="7.17319816s" totalImagesPullingTime="5.337932988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:15.172986667 +0000 UTC m=+61.368563690" watchObservedRunningTime="2026-07-07 11:41:15.17319816 +0000 UTC m=+61.368775183"
Jul 07 11:41:15.219324 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.219241    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7r6ns" podStartSLOduration=2.008520431 podStartE2EDuration="7.219224764s" totalImagesPullingTime="5.210704333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:15.219018645 +0000 UTC m=+61.414595680" watchObservedRunningTime="2026-07-07 11:41:15.219224764 +0000 UTC m=+61.414801786"
Jul 07 11:41:15.224385 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.224367    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"]
Jul 07 11:41:15.227342 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.227319    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.237738 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.237720    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 11:41:15.238056 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.238038    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 11:41:15.238293 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.238275    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 11:41:15.238404 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.238384    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 11:41:15.238490 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.238455    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wx5zg\""
Jul 07 11:41:15.238490 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.238482    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 11:41:15.267242 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.267221    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"]
Jul 07 11:41:15.272765 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.272746    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ea85072-891a-4c27-b309-af9dc0ffedaa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.272868 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.272807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ea85072-891a-4c27-b309-af9dc0ffedaa-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.272868 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.272827    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldp9h\" (UniqueName: \"kubernetes.io/projected/2ea85072-891a-4c27-b309-af9dc0ffedaa-kube-api-access-ldp9h\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.272967 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.272886    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea85072-891a-4c27-b309-af9dc0ffedaa-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.373648 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.373565    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ea85072-891a-4c27-b309-af9dc0ffedaa-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.373648 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.373610    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldp9h\" (UniqueName: \"kubernetes.io/projected/2ea85072-891a-4c27-b309-af9dc0ffedaa-kube-api-access-ldp9h\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.373648 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.373640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea85072-891a-4c27-b309-af9dc0ffedaa-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.373913 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.373704    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ea85072-891a-4c27-b309-af9dc0ffedaa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.374396 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.374367    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ea85072-891a-4c27-b309-af9dc0ffedaa-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.376237 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.376216    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ea85072-891a-4c27-b309-af9dc0ffedaa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.376503 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.376477    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea85072-891a-4c27-b309-af9dc0ffedaa-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.400235 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.400212    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldp9h\" (UniqueName: \"kubernetes.io/projected/2ea85072-891a-4c27-b309-af9dc0ffedaa-kube-api-access-ldp9h\") pod \"prometheus-operator-567f7b8b7c-8lqww\" (UID: \"2ea85072-891a-4c27-b309-af9dc0ffedaa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.416355 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.416327    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:15.536182 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.536156    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"
Jul 07 11:41:15.605354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.605333    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpspv"
Jul 07 11:41:15.658677 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.658583    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:15.688547 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.688517    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2tjsn" event={"ID":"b954c61a-ecae-42d2-9fd3-0c931f5cda90","Type":"ContainerStarted","Data":"9a383cdb973b7a7606e35ee3245b2014b12fe9291ecf07342e247e6cb1131372"}
Jul 07 11:41:15.693649 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.693608    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww"]
Jul 07 11:41:15.698463 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:15.698437    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea85072_891a_4c27_b309_af9dc0ffedaa.slice/crio-10661c7ef9dca80551c0eef4249dffd4b6b3a55da209c7f35d33422f4a827a2a WatchSource:0}: Error finding container 10661c7ef9dca80551c0eef4249dffd4b6b3a55da209c7f35d33422f4a827a2a: Status 404 returned error can't find the container with id 10661c7ef9dca80551c0eef4249dffd4b6b3a55da209c7f35d33422f4a827a2a
Jul 07 11:41:15.928343 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:15.928264    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:16.094854 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:16.094821    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:16.396737 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:16.396690    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:16.692524 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:16.692436    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww" event={"ID":"2ea85072-891a-4c27-b309-af9dc0ffedaa","Type":"ContainerStarted","Data":"10661c7ef9dca80551c0eef4249dffd4b6b3a55da209c7f35d33422f4a827a2a"}
Jul 07 11:41:16.761954 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:16.761924    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:18.532117 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:18.532077    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:18.701364 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:18.701329    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2tjsn" event={"ID":"b954c61a-ecae-42d2-9fd3-0c931f5cda90","Type":"ContainerStarted","Data":"44061e8f01900d175813c6a2a19180c1e8b7d3aea33ee854d9f6baec1d58bd96"}
Jul 07 11:41:18.702945 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:18.702919    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww" event={"ID":"2ea85072-891a-4c27-b309-af9dc0ffedaa","Type":"ContainerStarted","Data":"ea547a953c5e52f01f25445c49c8305f3f6442d8641b1f52e11db4b8b52af194"}
Jul 07 11:41:18.703102 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:18.702960    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww" event={"ID":"2ea85072-891a-4c27-b309-af9dc0ffedaa","Type":"ContainerStarted","Data":"34eb530ace733d9b270e1e72dc82aa001d331c42d29500836a2eaec73856eb82"}
Jul 07 11:41:18.749725 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:18.749582    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2tjsn" podStartSLOduration=1.859347701 podStartE2EDuration="10.749565202s" totalImagesPullingTime="8.890217501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:41:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:18.748926275 +0000 UTC m=+64.944503299" watchObservedRunningTime="2026-07-07 11:41:18.749565202 +0000 UTC m=+64.945142227"
Jul 07 11:41:20.009139 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.009105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:41:20.020458 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.020437    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 11:41:20.032101 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.032077    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66a5788d-0db4-4335-ae38-7e916eca3b86-original-pull-secret\") pod \"global-pull-secret-syncer-wrrg9\" (UID: \"66a5788d-0db4-4335-ae38-7e916eca3b86\") " pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:41:20.109357 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.109332    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:41:20.112631 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.112612    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 11:41:20.121754 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.121730    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d622f883-4224-4801-b1d3-2b543c30dc35-metrics-certs\") pod \"network-metrics-daemon-nwrkz\" (UID: \"d622f883-4224-4801-b1d3-2b543c30dc35\") " pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:41:20.210280 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.210254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:20.213049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.213031    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 11:41:20.223895 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.223872    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 11:41:20.233672 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.233644    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/0971da9a-0556-4091-a7de-253ec8dce3e4-kube-api-access-fm8rq\") pod \"network-check-target-zzbqh\" (UID: \"0971da9a-0556-4091-a7de-253ec8dce3e4\") " pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:20.250302 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.250281    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrrg9"
Jul 07 11:41:20.269798 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.269748    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5vsqg\""
Jul 07 11:41:20.273018 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.273005    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z7ksr\""
Jul 07 11:41:20.277096 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.277081    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:20.279285 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.279270    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwrkz"
Jul 07 11:41:20.478329 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.478244    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8lqww" podStartSLOduration=3.227057847 podStartE2EDuration="5.47822617s" totalImagesPullingTime="2.251168323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:18.822500067 +0000 UTC m=+65.018077089" watchObservedRunningTime="2026-07-07 11:41:20.47822617 +0000 UTC m=+66.673803192"
Jul 07 11:41:20.478923 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.478852    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wrrg9"]
Jul 07 11:41:20.483264 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.481389    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zzbqh"]
Jul 07 11:41:20.519329 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:20.519308    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd622f883_4224_4801_b1d3_2b543c30dc35.slice/crio-381d64824ea93b191ddcb3988a1499796ec0f72eddf27ab1d27ce921d8731e8b WatchSource:0}: Error finding container 381d64824ea93b191ddcb3988a1499796ec0f72eddf27ab1d27ce921d8731e8b: Status 404 returned error can't find the container with id 381d64824ea93b191ddcb3988a1499796ec0f72eddf27ab1d27ce921d8731e8b
Jul 07 11:41:20.521774 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.521757    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwrkz"]
Jul 07 11:41:20.555769 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.555738    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:20.709956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.709931    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zzbqh" event={"ID":"0971da9a-0556-4091-a7de-253ec8dce3e4","Type":"ContainerStarted","Data":"8e9444006f65fdeaae44444a5e67d041ec7270232ae21785dc50ca835a73f3b2"}
Jul 07 11:41:20.710965 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.710935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwrkz" event={"ID":"d622f883-4224-4801-b1d3-2b543c30dc35","Type":"ContainerStarted","Data":"381d64824ea93b191ddcb3988a1499796ec0f72eddf27ab1d27ce921d8731e8b"}
Jul 07 11:41:20.711922 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.711904    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wrrg9" event={"ID":"66a5788d-0db4-4335-ae38-7e916eca3b86","Type":"ContainerStarted","Data":"fe154cd218f2b339e6933e84446870853805d478365a41a41c4a31d6f7770d60"}
Jul 07 11:41:20.922365 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:20.922280    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:21.270900 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.270866    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:21.532986 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.532906    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:21.757185 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.757160    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"]
Jul 07 11:41:21.759863 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.759833    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.821881 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.821858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5p7m\" (UniqueName: \"kubernetes.io/projected/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-api-access-h5p7m\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.822077 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.821901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.822077 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.821946    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4057a7ba-08cd-41d6-852e-25500d526c6f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.822077 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.822003    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.822077 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.822031    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.822077 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.822053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.856201 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:21.855747    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-tls\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret"
Jul 07 11:41:21.867456 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:21.867418    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"kube-state-metrics-7c9dbb8b7d-lrrbt\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" podUID="4057a7ba-08cd-41d6-852e-25500d526c6f" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.868803 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:21.868769    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-dockercfg-hpmbm\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hpmbm\"" type="*v1.Secret"
Jul 07 11:41:21.868903 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:21.868849    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret"
Jul 07 11:41:21.870303 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:21.870276    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-state-metrics-custom-resource-state-configmap\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap"
Jul 07 11:41:21.922937 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.922907    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4057a7ba-08cd-41d6-852e-25500d526c6f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.923054 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.922962    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.923054 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.922994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.923054 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.923022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.923209 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.923056    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5p7m\" (UniqueName: \"kubernetes.io/projected/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-api-access-h5p7m\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.923209 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.923086    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.923384 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.923365    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4057a7ba-08cd-41d6-852e-25500d526c6f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.923709 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.923645    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:21.935044 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.934980    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:21.991712 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:21.991684    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"]
Jul 07 11:41:22.064551 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.064483    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n99nd"]
Jul 07 11:41:22.066931 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.066909    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.081152 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:22.081118    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-n99nd\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" podUID="eec050d3-2cc3-4dca-82cd-e10bac52d313" pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.082384 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.082364    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 11:41:22.082959 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:22.082927    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret"
Jul 07 11:41:22.083800 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.083611    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gktvm\""
Jul 07 11:41:22.100694 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.100668    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5p7m\" (UniqueName: \"kubernetes.io/projected/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-api-access-h5p7m\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:22.112602 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.112578    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 11:41:22.124791 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.124767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-wtmp\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.124894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.124814    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-sys\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.124894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.124839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-tls\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.124894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.124865    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eec050d3-2cc3-4dca-82cd-e10bac52d313-metrics-client-ca\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.124894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.124892    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-root\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.125166 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.124913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-textfile\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.125166 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.124946    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wkj4\" (UniqueName: \"kubernetes.io/projected/eec050d3-2cc3-4dca-82cd-e10bac52d313-kube-api-access-8wkj4\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.125166 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.125014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.125166 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.125056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-accelerators-collector-config\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.225821 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.225790    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.225964 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.225838    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-accelerators-collector-config\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.225964 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.225871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-wtmp\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226109 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226089    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-sys\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226168 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-tls\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226168 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226134    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-wtmp\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226168 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226147    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eec050d3-2cc3-4dca-82cd-e10bac52d313-metrics-client-ca\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226313 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226196    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-root\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226313 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226225    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-textfile\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226313 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wkj4\" (UniqueName: \"kubernetes.io/projected/eec050d3-2cc3-4dca-82cd-e10bac52d313-kube-api-access-8wkj4\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226454 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226322    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-root\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226454 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226442    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eec050d3-2cc3-4dca-82cd-e10bac52d313-sys\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.226779 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.226755    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-accelerators-collector-config\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.227092 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.227074    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-textfile\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.227150 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.227136    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eec050d3-2cc3-4dca-82cd-e10bac52d313-metrics-client-ca\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.228809 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.228784    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.286900 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.286876    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wkj4\" (UniqueName: \"kubernetes.io/projected/eec050d3-2cc3-4dca-82cd-e10bac52d313-kube-api-access-8wkj4\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:22.609810 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.609704    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:22.673920 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.673890    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 11:41:22.686832 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.686806    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:22.723010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.722982    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwrkz" event={"ID":"d622f883-4224-4801-b1d3-2b543c30dc35","Type":"ContainerStarted","Data":"c8801e5d89bc110af9bf74e1778bdc0c37eebd777332a3ab2b2058e79686be0f"}
Jul 07 11:41:22.723152 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.723040    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwrkz" event={"ID":"d622f883-4224-4801-b1d3-2b543c30dc35","Type":"ContainerStarted","Data":"312407f603d79730cd65ff6294427749f47ebf7ea9fbd92814d5593bcb51edf5"}
Jul 07 11:41:22.824754 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:22.824697    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nwrkz" podStartSLOduration=67.737284018 podStartE2EDuration="1m8.824635994s" totalImagesPullingTime="1.087351976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:22.823986859 +0000 UTC m=+69.019563883" watchObservedRunningTime="2026-07-07 11:41:22.824635994 +0000 UTC m=+69.020213017"
Jul 07 11:41:22.924139 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:22.924064    2572 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition
Jul 07 11:41:22.924272 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:22.924066    2572 configmap.go:193] Couldn't get configMap openshift-monitoring/kube-state-metrics-custom-resource-state-configmap: failed to sync configmap cache: timed out waiting for the condition
Jul 07 11:41:22.924272 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:22.924165    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-kube-rbac-proxy-config podName:4057a7ba-08cd-41d6-852e-25500d526c6f nodeName:}" failed. No retries permitted until 2026-07-07 11:41:23.42414222 +0000 UTC m=+69.619719235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-kube-rbac-proxy-config") pod "kube-state-metrics-7c9dbb8b7d-lrrbt" (UID: "4057a7ba-08cd-41d6-852e-25500d526c6f") : failed to sync secret cache: timed out waiting for the condition
Jul 07 11:41:22.924272 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:41:22.924264    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-custom-resource-state-configmap podName:4057a7ba-08cd-41d6-852e-25500d526c6f nodeName:}" failed. No retries permitted until 2026-07-07 11:41:23.424244905 +0000 UTC m=+69.619821910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-custom-resource-state-configmap" (UniqueName: "kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-custom-resource-state-configmap") pod "kube-state-metrics-7c9dbb8b7d-lrrbt" (UID: "4057a7ba-08cd-41d6-852e-25500d526c6f") : failed to sync configmap cache: timed out waiting for the condition
Jul 07 11:41:23.120135 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.120107    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 11:41:23.130792 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.130766    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eec050d3-2cc3-4dca-82cd-e10bac52d313-node-exporter-tls\") pod \"node-exporter-n99nd\" (UID: \"eec050d3-2cc3-4dca-82cd-e10bac52d313\") " pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:23.180338 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.180159    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:23.202899 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.202873    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 11:41:23.236405 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.236382    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 11:41:23.278242 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.278211    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n99nd"
Jul 07 11:41:23.353881 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.353854    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hpmbm\""
Jul 07 11:41:23.436773 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.436700    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:23.436773 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.436745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:23.437527 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.437499    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:23.439616 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.439588    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4057a7ba-08cd-41d6-852e-25500d526c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-lrrbt\" (UID: \"4057a7ba-08cd-41d6-852e-25500d526c6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:23.462151 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.462035    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:23.572088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.572055    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"
Jul 07 11:41:23.739068 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:23.738925    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:24.691024 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:24.690982    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7r6ns"
Jul 07 11:41:24.902268 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:24.901492    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec050d3_2cc3_4dca_82cd_e10bac52d313.slice/crio-c8c1496f61ef987b3dcb650716ec861dc5c6eaf0d3fa732922547a21d8cd927c WatchSource:0}: Error finding container c8c1496f61ef987b3dcb650716ec861dc5c6eaf0d3fa732922547a21d8cd927c: Status 404 returned error can't find the container with id c8c1496f61ef987b3dcb650716ec861dc5c6eaf0d3fa732922547a21d8cd927c
Jul 07 11:41:25.057030 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.056895    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt"]
Jul 07 11:41:25.616072 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.616041    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:25.712740 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.712716    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"]
Jul 07 11:41:25.715807 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.715788    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:25.723315 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.723296    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 11:41:25.723620 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.723599    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-672kl\""
Jul 07 11:41:25.739217 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.739189    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wrrg9" event={"ID":"66a5788d-0db4-4335-ae38-7e916eca3b86","Type":"ContainerStarted","Data":"8f5efae307ddc98f7f94e3854534c5194514406fbb535110fb941d9a0a74ce31"}
Jul 07 11:41:25.740529 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.740504    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt" event={"ID":"4057a7ba-08cd-41d6-852e-25500d526c6f","Type":"ContainerStarted","Data":"169ab65558af1af5d3be8107d2bb7ee275c1abe52882c4f41de166817a1219b9"}
Jul 07 11:41:25.742360 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.742217    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zzbqh" event={"ID":"0971da9a-0556-4091-a7de-253ec8dce3e4","Type":"ContainerStarted","Data":"e3d3f8d9fdcaf8ce311b47db0e57bf8f2c1e80436e6bda1f1b6f091cfd4e56ef"}
Jul 07 11:41:25.742360 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.742275    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:41:25.743517 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.743493    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n99nd" event={"ID":"eec050d3-2cc3-4dca-82cd-e10bac52d313","Type":"ContainerStarted","Data":"c8c1496f61ef987b3dcb650716ec861dc5c6eaf0d3fa732922547a21d8cd927c"}
Jul 07 11:41:25.788089 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.788007    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"]
Jul 07 11:41:25.857158 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.856954    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gltcc\" (UniqueName: \"kubernetes.io/projected/9a228e1c-da62-4d64-ae7c-70999bf0e97e-kube-api-access-gltcc\") pod \"monitoring-plugin-d5dcff7cf-k87rg\" (UID: \"9a228e1c-da62-4d64-ae7c-70999bf0e97e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:25.857158 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.857002    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a228e1c-da62-4d64-ae7c-70999bf0e97e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k87rg\" (UID: \"9a228e1c-da62-4d64-ae7c-70999bf0e97e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:25.958375 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.958311    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gltcc\" (UniqueName: \"kubernetes.io/projected/9a228e1c-da62-4d64-ae7c-70999bf0e97e-kube-api-access-gltcc\") pod \"monitoring-plugin-d5dcff7cf-k87rg\" (UID: \"9a228e1c-da62-4d64-ae7c-70999bf0e97e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:25.958375 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.958344    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a228e1c-da62-4d64-ae7c-70999bf0e97e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k87rg\" (UID: \"9a228e1c-da62-4d64-ae7c-70999bf0e97e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:25.961234 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.961212    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a228e1c-da62-4d64-ae7c-70999bf0e97e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k87rg\" (UID: \"9a228e1c-da62-4d64-ae7c-70999bf0e97e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:25.998341 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:25.998320    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gltcc\" (UniqueName: \"kubernetes.io/projected/9a228e1c-da62-4d64-ae7c-70999bf0e97e-kube-api-access-gltcc\") pod \"monitoring-plugin-d5dcff7cf-k87rg\" (UID: \"9a228e1c-da62-4d64-ae7c-70999bf0e97e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:26.026288 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.026265    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:26.037259 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.036998    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wrrg9" podStartSLOduration=67.610132876 podStartE2EDuration="1m12.036981051s" totalImagesPullingTime="4.426848175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:26.036221571 +0000 UTC m=+72.231798594" watchObservedRunningTime="2026-07-07 11:41:26.036981051 +0000 UTC m=+72.232558075"
Jul 07 11:41:26.102061 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.102006    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:26.106476 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.106432    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zzbqh" podStartSLOduration=67.688227119 podStartE2EDuration="1m12.106416712s" totalImagesPullingTime="4.418189593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:26.106173085 +0000 UTC m=+72.301750107" watchObservedRunningTime="2026-07-07 11:41:26.106416712 +0000 UTC m=+72.301993771"
Jul 07 11:41:26.291218 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.291181    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"]
Jul 07 11:41:26.382898 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:26.382870    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a228e1c_da62_4d64_ae7c_70999bf0e97e.slice/crio-306696971e18599080a930d66c723a6b647de53b8db17e8711a6dde616e1adb0 WatchSource:0}: Error finding container 306696971e18599080a930d66c723a6b647de53b8db17e8711a6dde616e1adb0: Status 404 returned error can't find the container with id 306696971e18599080a930d66c723a6b647de53b8db17e8711a6dde616e1adb0
Jul 07 11:41:26.430024 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.429993    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:26.544878 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.544856    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"]
Jul 07 11:41:26.548197 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.548176    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.574092 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.574074    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 11:41:26.574092 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.574082    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 11:41:26.575420 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.575402    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 11:41:26.576136 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.576118    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 11:41:26.576732 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.576712    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6smsx\""
Jul 07 11:41:26.583863 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.583844    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 11:41:26.584076 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.584064    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9i7nhphss5fv\""
Jul 07 11:41:26.663760 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.663735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fc39e06-3f4d-40bd-89d2-32af355174ac-metrics-client-ca\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.663916 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.663771    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.663916 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.663801    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-tls\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.663916 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.663871    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.664034 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.663919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-grpc-tls\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.664034 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.663941    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.664034 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.663960    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.664034 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.664010    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgcxb\" (UniqueName: \"kubernetes.io/projected/4fc39e06-3f4d-40bd-89d2-32af355174ac-kube-api-access-sgcxb\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.747720 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.747683    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg" event={"ID":"9a228e1c-da62-4d64-ae7c-70999bf0e97e","Type":"ContainerStarted","Data":"306696971e18599080a930d66c723a6b647de53b8db17e8711a6dde616e1adb0"}
Jul 07 11:41:26.749727 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.749696    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt" event={"ID":"4057a7ba-08cd-41d6-852e-25500d526c6f","Type":"ContainerStarted","Data":"db3bacf9c881076b33bfe87eb4de349b4ef02c6c50016e5478ee59996f12288c"}
Jul 07 11:41:26.749829 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.749750    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt" event={"ID":"4057a7ba-08cd-41d6-852e-25500d526c6f","Type":"ContainerStarted","Data":"c2285d4c80374f2ac823697158ec81e278827a2da1f18e4c64b0102afd3ce4d3"}
Jul 07 11:41:26.749829 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.749781    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt" event={"ID":"4057a7ba-08cd-41d6-852e-25500d526c6f","Type":"ContainerStarted","Data":"feadb17e3c35c683134e503acacb0e8d13502b285d8ec66bfabd50c4e247f59f"}
Jul 07 11:41:26.751227 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.751203    2572 generic.go:356] "Generic (PLEG): container finished" podID="eec050d3-2cc3-4dca-82cd-e10bac52d313" containerID="4ecffa0bd8691128be43ba4b4a2ab07e956e2f748e98637f0ab1fc8fb9f8fccd" exitCode=0
Jul 07 11:41:26.751296 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.751271    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n99nd" event={"ID":"eec050d3-2cc3-4dca-82cd-e10bac52d313","Type":"ContainerDied","Data":"4ecffa0bd8691128be43ba4b4a2ab07e956e2f748e98637f0ab1fc8fb9f8fccd"}
Jul 07 11:41:26.765273 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.765253    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fc39e06-3f4d-40bd-89d2-32af355174ac-metrics-client-ca\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.765373 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.765281    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.765373 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.765299    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-tls\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.765473 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.765433    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.765528 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.765472    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-grpc-tls\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.765528 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.765506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.765630 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.765548    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.765630 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.765583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgcxb\" (UniqueName: \"kubernetes.io/projected/4fc39e06-3f4d-40bd-89d2-32af355174ac-kube-api-access-sgcxb\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.766157 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.766104    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fc39e06-3f4d-40bd-89d2-32af355174ac-metrics-client-ca\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.767967 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.767937    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.768212 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.768188    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.768347 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.768325    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.768391 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.768335    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-tls\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.768546 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.768532    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:26.768641 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:26.768626    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fc39e06-3f4d-40bd-89d2-32af355174ac-secret-grpc-tls\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:27.013545 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:27.013516    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgcxb\" (UniqueName: \"kubernetes.io/projected/4fc39e06-3f4d-40bd-89d2-32af355174ac-kube-api-access-sgcxb\") pod \"thanos-querier-78db8f4c79-rm4kf\" (UID: \"4fc39e06-3f4d-40bd-89d2-32af355174ac\") " pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:27.159267 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:27.159191    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:27.221771 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:27.221708    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"]
Jul 07 11:41:27.284478 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:27.284426    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lrrbt" podStartSLOduration=4.923183994 podStartE2EDuration="6.284409547s" totalImagesPullingTime="1.361225553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:27.208790009 +0000 UTC m=+73.404367033" watchObservedRunningTime="2026-07-07 11:41:27.284409547 +0000 UTC m=+73.479986569"
Jul 07 11:41:27.591465 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:27.591435    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"]
Jul 07 11:41:27.656776 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:27.656746    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc39e06_3f4d_40bd_89d2_32af355174ac.slice/crio-c4b5d24d4e39c4a8d3d3fdd916c48489420176725696216692059499fd39823b WatchSource:0}: Error finding container c4b5d24d4e39c4a8d3d3fdd916c48489420176725696216692059499fd39823b: Status 404 returned error can't find the container with id c4b5d24d4e39c4a8d3d3fdd916c48489420176725696216692059499fd39823b
Jul 07 11:41:27.757053 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:27.757023    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf" event={"ID":"4fc39e06-3f4d-40bd-89d2-32af355174ac","Type":"ContainerStarted","Data":"c4b5d24d4e39c4a8d3d3fdd916c48489420176725696216692059499fd39823b"}
Jul 07 11:41:27.759267 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:27.759241    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n99nd" event={"ID":"eec050d3-2cc3-4dca-82cd-e10bac52d313","Type":"ContainerStarted","Data":"460c96f5fcb079f3eb8f0d6aed661ffacd98e24b0eb26d8a10a288e5b0b2493f"}
Jul 07 11:41:27.759375 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:27.759291    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n99nd" event={"ID":"eec050d3-2cc3-4dca-82cd-e10bac52d313","Type":"ContainerStarted","Data":"cd2b55400ec31a7c885279842f122e79d161ff5b54658b3e41723d49938cc130"}
Jul 07 11:41:28.100697 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:28.100652    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:28.368598 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:28.368518    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:28.766225 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:28.766185    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg" event={"ID":"9a228e1c-da62-4d64-ae7c-70999bf0e97e","Type":"ContainerStarted","Data":"7367ffb0cf29502df016bc2284775aef7f38fd403a84b35216a6e14cc6481960"}
Jul 07 11:41:28.766653 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:28.766353    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:28.772967 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:28.772907    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg"
Jul 07 11:41:28.820788 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:28.820743    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n99nd" podStartSLOduration=6.946311476 podStartE2EDuration="7.820727345s" totalImagesPullingTime="800.910227ms" totalInitContainerRuntime="73.505642ms" isStatefulPod=true podCreationTimestamp="2026-07-07 11:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:27.907894606 +0000 UTC m=+74.103471639" watchObservedRunningTime="2026-07-07 11:41:28.820727345 +0000 UTC m=+75.016304368"
Jul 07 11:41:28.820947 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:28.820844    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k87rg" podStartSLOduration=2.504057362 podStartE2EDuration="3.820840051s" totalImagesPullingTime="1.316782689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:28.820110105 +0000 UTC m=+75.015687129" watchObservedRunningTime="2026-07-07 11:41:28.820840051 +0000 UTC m=+75.016417072"
Jul 07 11:41:29.531241 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.531201    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:29.555562 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.555539    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:41:29.560545 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.560527    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.584037 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.583891    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 11:41:29.584037 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.583937    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 11:41:29.674269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.674178    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 11:41:29.674411 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.674294    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 11:41:29.674821 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.674782    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 11:41:29.675598 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.675265    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 11:41:29.675598 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.675280    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 11:41:29.675598 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.675470    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zkm6m\""
Jul 07 11:41:29.677329 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.676755    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 11:41:29.677329 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.676972    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 11:41:29.677329 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.677004    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 11:41:29.677329 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.677032    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9e2oirm9slk82\""
Jul 07 11:41:29.677572 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.677505    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 11:41:29.677981 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.677753    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 11:41:29.679033 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.678178    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 11:41:29.681266 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.681247    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 11:41:29.683306 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.683284    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 11:41:29.691964 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.691941    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692056 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.691979    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692056 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692134 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692070    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692134 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692087    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692134 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692113    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692134 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692132    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-config\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692251 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692146    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncbgr\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-kube-api-access-ncbgr\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692251 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-web-config\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692251 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692195    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692251 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692251 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692229    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-config-out\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692251 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692243    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692408 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692408 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692294    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692408 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692408 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692359    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692408 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692394    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692553 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692431    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.692553 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.692465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.717179 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.717158    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:41:29.771501 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.771473    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf" event={"ID":"4fc39e06-3f4d-40bd-89d2-32af355174ac","Type":"ContainerStarted","Data":"62a1d418e85f890df16d537dafa8c9c713818e99de652b0cd454f0587bd9670c"}
Jul 07 11:41:29.771835 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.771548    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf" event={"ID":"4fc39e06-3f4d-40bd-89d2-32af355174ac","Type":"ContainerStarted","Data":"7a4cf80496237465def3ad0fac9916dfb4fb70048ff96e1c0e07b2d34933a9e3"}
Jul 07 11:41:29.771835 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.771578    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf" event={"ID":"4fc39e06-3f4d-40bd-89d2-32af355174ac","Type":"ContainerStarted","Data":"3f35093b01a394c0cae8e23a6e32c7d85494868d11069884af75e1aaefcf6532"}
Jul 07 11:41:29.794950 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.794927    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.795124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.794967    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.795124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.794987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.795124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.795124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795041    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.795124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795069    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.795124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795087    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.795124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795148    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-config\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795172    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncbgr\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-kube-api-access-ncbgr\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-web-config\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795473    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-config-out\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795542    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795568    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795595    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795600    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795628    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.795953    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.796879 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.796353    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.797028 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.797001    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.797569 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.797542    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.798194 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.798168    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.798375 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.798354    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.798558 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.798535    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.798757 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.798706    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.799268 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.799242    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.799629 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.799576    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.800273 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.800224    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.801273 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.801246    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.801456 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.801411    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.801551 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.801486    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-config-out\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.801894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.801870    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-config\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.802099 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.802065    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.802168 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.802116    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.802385 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.802368    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.802554 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.802538    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-web-config\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.839721 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.831847    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncbgr\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-kube-api-access-ncbgr\") pod \"prometheus-k8s-0\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.874379 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:29.874347    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:30.056583 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.056422    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:41:30.058837 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:41:30.058808    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff454b5_56b8_42d1_b8fe_e56c355e4519.slice/crio-40ad999abeb17f2c52a3ab9a236f5584a8e48391a5ed3ac44cc858e5846f9c1e WatchSource:0}: Error finding container 40ad999abeb17f2c52a3ab9a236f5584a8e48391a5ed3ac44cc858e5846f9c1e: Status 404 returned error can't find the container with id 40ad999abeb17f2c52a3ab9a236f5584a8e48391a5ed3ac44cc858e5846f9c1e
Jul 07 11:41:30.780002 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.779965    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf" event={"ID":"4fc39e06-3f4d-40bd-89d2-32af355174ac","Type":"ContainerStarted","Data":"ecdf113f16d01256adbb320e5a0a586cc4f1e51446e6b61a07f70aeb58557997"}
Jul 07 11:41:30.780456 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.780049    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf" event={"ID":"4fc39e06-3f4d-40bd-89d2-32af355174ac","Type":"ContainerStarted","Data":"88d4e4edaba72eecad9d1bcb5eba25493002c209ea368d750a870936549295b8"}
Jul 07 11:41:30.780456 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.780091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf" event={"ID":"4fc39e06-3f4d-40bd-89d2-32af355174ac","Type":"ContainerStarted","Data":"a7f3bf1d58645a25150e83ce884b11e71b13fce70cf1556750415c8c5aa99442"}
Jul 07 11:41:30.780456 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.780135    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:30.781409 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.781384    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerStarted","Data":"40ad999abeb17f2c52a3ab9a236f5584a8e48391a5ed3ac44cc858e5846f9c1e"}
Jul 07 11:41:30.784402 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.784344    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:30.846568 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.846514    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf" podStartSLOduration=2.387868712 podStartE2EDuration="4.846493188s" totalImagesPullingTime="2.458624476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:30.842244576 +0000 UTC m=+77.037821598" watchObservedRunningTime="2026-07-07 11:41:30.846493188 +0000 UTC m=+77.042070213"
Jul 07 11:41:30.991909 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:30.991864    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:31.787559 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:31.787526    2572 generic.go:356] "Generic (PLEG): container finished" podID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e" exitCode=0
Jul 07 11:41:31.787999 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:31.787610    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerDied","Data":"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"}
Jul 07 11:41:32.177004 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:32.176930    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:32.355606 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:32.355573    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:32.653721 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:32.653674    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:32.798741 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:32.798704    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:33.442590 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:33.442556    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:33.646157 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:33.646113    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:33.867047 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:33.866995    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:34.073274 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:34.073238    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:34.270579 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:34.270538    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:34.477881 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:34.477786    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:34.802959 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:34.802935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerStarted","Data":"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"}
Jul 07 11:41:34.803057 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:34.803042    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerStarted","Data":"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"}
Jul 07 11:41:35.104043 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.104011    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:35.215351 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.215287    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:35.411211 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.411172    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:35.563591 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.563559    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:35.810147 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.810121    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerStarted","Data":"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"}
Jul 07 11:41:35.810262 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.810193    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerStarted","Data":"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"}
Jul 07 11:41:35.810262 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.810242    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerStarted","Data":"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"}
Jul 07 11:41:35.810381 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.810281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerStarted","Data":"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"}
Jul 07 11:41:35.911920 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:35.911830    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9930924279999997 podStartE2EDuration="6.911815369s" totalImagesPullingTime="3.896084866s" totalInitContainerRuntime="22.638075ms" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:35.901213349 +0000 UTC m=+82.096790371" watchObservedRunningTime="2026-07-07 11:41:35.911815369 +0000 UTC m=+82.107392416"
Jul 07 11:41:36.323383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:36.323348    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:36.478811 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:36.478786    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:36.794012 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:36.793993    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78db8f4c79-rm4kf"
Jul 07 11:41:39.874847 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:39.874815    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:40.656077 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:40.656031    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:40.730124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:40.730091    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:45.536999 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:45.536965    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:45.810557 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:45.810485    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:47.726892 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:47.726851    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:47.860950 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:47.860913    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:50.367942 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:50.367896    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:50.443377 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:50.443340    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:54.292299 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:54.292263    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:41:56.753616 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:41:56.753587    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zzbqh"
Jul 07 11:42:01.247751 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:01.247711    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:01.317248 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:01.317220    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:05.391569 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:05.391531    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:05.502983 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:05.502953    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:07.673165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:07.673131    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:07.773810 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:07.773778    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:10.053503 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:10.053461    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:11.944081 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:11.944046    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:20.354947 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:20.354908    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:20.415234 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:20.415197    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:20.786081 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:20.786052    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:20.858793 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:20.858762    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:29.874744 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:29.874712    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:29.893668 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:29.893622    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:30.017991 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:30.017969    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:42.317558 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:42.317520    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:42.523638 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:42.523597    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:47.709770 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:47.709732    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:47.710267 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:47.710149    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="prometheus" containerID="cri-o://209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06" gracePeriod=600
Jul 07 11:42:47.710267 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:47.710201    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="config-reloader" containerID="cri-o://0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7" gracePeriod=600
Jul 07 11:42:47.710267 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:47.710255    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy-thanos" containerID="cri-o://09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6" gracePeriod=600
Jul 07 11:42:47.710451 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:47.710286    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy-web" containerID="cri-o://c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665" gracePeriod=600
Jul 07 11:42:47.710451 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:47.710211    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="thanos-sidecar" containerID="cri-o://a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6" gracePeriod=600
Jul 07 11:42:47.710451 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:47.710364    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy" containerID="cri-o://d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5" gracePeriod=600
Jul 07 11:42:47.981143 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:47.981118    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.054677 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054641    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-config-out\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.054811 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054696    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-serving-certs-ca-bundle\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.054811 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054726    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-config\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.054811 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054763    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-db\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.054811 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054793    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-trusted-ca-bundle\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054816    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-metrics-client-certs\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054845    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-tls\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054872    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-thanos-prometheus-http-client-file\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054923    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-kubelet-serving-ca-bundle\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054950    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-web-config\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.054977    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-grpc-tls\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055005    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-metrics-client-ca\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055049 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055038    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055469 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055070    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-2\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055469 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055100    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:48.055469 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055133    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncbgr\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-kube-api-access-ncbgr\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055469 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055162    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-1\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055469 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055196    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-tls-assets\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055469 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055225    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-kube-rbac-proxy\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055469 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055256    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055469 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055284    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-0\") pod \"cff454b5-56b8-42d1-b8fe-e56c355e4519\" (UID: \"cff454b5-56b8-42d1-b8fe-e56c355e4519\") "
Jul 07 11:42:48.055889 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055507    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.055889 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055804    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:48.055889 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.055861    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:42:48.056622 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.056576    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:48.056785 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.056635    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:48.057057 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.057036    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:48.057452 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.057431    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:48.057547 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.057495    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-config-out" (OuterVolumeSpecName: "config-out") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:42:48.057630 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.057606    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:48.057813 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.057766    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-config" (OuterVolumeSpecName: "config") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.058497 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.058465    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.059331 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.059297    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.059614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.059572    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.059868 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.059833    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.060121 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.060060    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.060121 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.060092    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.060275 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.060254    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:42:48.060440 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.060406    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-kube-api-access-ncbgr" (OuterVolumeSpecName: "kube-api-access-ncbgr") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "kube-api-access-ncbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:42:48.061488 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.061467    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.070515 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.070494    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-web-config" (OuterVolumeSpecName: "web-config") pod "cff454b5-56b8-42d1-b8fe-e56c355e4519" (UID: "cff454b5-56b8-42d1-b8fe-e56c355e4519"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:48.074400 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074381    2572 generic.go:356] "Generic (PLEG): container finished" podID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6" exitCode=0
Jul 07 11:42:48.074400 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074398    2572 generic.go:356] "Generic (PLEG): container finished" podID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5" exitCode=0
Jul 07 11:42:48.074505 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074404    2572 generic.go:356] "Generic (PLEG): container finished" podID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665" exitCode=0
Jul 07 11:42:48.074505 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074410    2572 generic.go:356] "Generic (PLEG): container finished" podID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6" exitCode=0
Jul 07 11:42:48.074505 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074416    2572 generic.go:356] "Generic (PLEG): container finished" podID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7" exitCode=0
Jul 07 11:42:48.074505 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074421    2572 generic.go:356] "Generic (PLEG): container finished" podID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06" exitCode=0
Jul 07 11:42:48.074505 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074458    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.074505 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerDied","Data":"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"}
Jul 07 11:42:48.074691 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074522    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerDied","Data":"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"}
Jul 07 11:42:48.074691 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074549    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerDied","Data":"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"}
Jul 07 11:42:48.074691 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074575    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerDied","Data":"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"}
Jul 07 11:42:48.074691 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074607    2572 scope.go:117] "RemoveContainer" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"
Jul 07 11:42:48.074691 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074619    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerDied","Data":"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"}
Jul 07 11:42:48.074691 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074643    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerDied","Data":"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"}
Jul 07 11:42:48.074881 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.074711    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cff454b5-56b8-42d1-b8fe-e56c355e4519","Type":"ContainerDied","Data":"40ad999abeb17f2c52a3ab9a236f5584a8e48391a5ed3ac44cc858e5846f9c1e"}
Jul 07 11:42:48.082894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.082869    2572 scope.go:117] "RemoveContainer" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"
Jul 07 11:42:48.089325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.089311    2572 scope.go:117] "RemoveContainer" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"
Jul 07 11:42:48.095640 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.095621    2572 scope.go:117] "RemoveContainer" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"
Jul 07 11:42:48.099024 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.098810    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.102870 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.102848    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.104698 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.104649    2572 scope.go:117] "RemoveContainer" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"
Jul 07 11:42:48.111213 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.111192    2572 scope.go:117] "RemoveContainer" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"
Jul 07 11:42:48.117646 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.117630    2572 scope.go:117] "RemoveContainer" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"
Jul 07 11:42:48.123872 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.123858    2572 scope.go:117] "RemoveContainer" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"
Jul 07 11:42:48.124104 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:42:48.124088    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": container with ID starting with 09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6 not found: ID does not exist" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"
Jul 07 11:42:48.124145 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.124111    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"} err="failed to get container status \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": rpc error: code = NotFound desc = could not find container \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": container with ID starting with 09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6 not found: ID does not exist"
Jul 07 11:42:48.124145 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.124139    2572 scope.go:117] "RemoveContainer" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"
Jul 07 11:42:48.124349 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:42:48.124334    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": container with ID starting with d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5 not found: ID does not exist" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"
Jul 07 11:42:48.124391 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.124353    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"} err="failed to get container status \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": rpc error: code = NotFound desc = could not find container \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": container with ID starting with d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5 not found: ID does not exist"
Jul 07 11:42:48.124391 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.124368    2572 scope.go:117] "RemoveContainer" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"
Jul 07 11:42:48.124550 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:42:48.124536    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": container with ID starting with c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665 not found: ID does not exist" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"
Jul 07 11:42:48.124588 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.124553    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"} err="failed to get container status \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": rpc error: code = NotFound desc = could not find container \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": container with ID starting with c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665 not found: ID does not exist"
Jul 07 11:42:48.124588 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.124565    2572 scope.go:117] "RemoveContainer" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"
Jul 07 11:42:48.124869 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:42:48.124852    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": container with ID starting with a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6 not found: ID does not exist" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"
Jul 07 11:42:48.124939 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.124877    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"} err="failed to get container status \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": rpc error: code = NotFound desc = could not find container \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": container with ID starting with a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6 not found: ID does not exist"
Jul 07 11:42:48.124939 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.124899    2572 scope.go:117] "RemoveContainer" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"
Jul 07 11:42:48.125169 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:42:48.125153    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": container with ID starting with 0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7 not found: ID does not exist" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"
Jul 07 11:42:48.125210 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.125181    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"} err="failed to get container status \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": rpc error: code = NotFound desc = could not find container \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": container with ID starting with 0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7 not found: ID does not exist"
Jul 07 11:42:48.125210 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.125194    2572 scope.go:117] "RemoveContainer" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"
Jul 07 11:42:48.125408 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:42:48.125395    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": container with ID starting with 209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06 not found: ID does not exist" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"
Jul 07 11:42:48.125448 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.125412    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"} err="failed to get container status \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": rpc error: code = NotFound desc = could not find container \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": container with ID starting with 209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06 not found: ID does not exist"
Jul 07 11:42:48.125448 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.125425    2572 scope.go:117] "RemoveContainer" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"
Jul 07 11:42:48.125608 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:42:48.125594    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": container with ID starting with e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e not found: ID does not exist" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"
Jul 07 11:42:48.125644 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.125611    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"} err="failed to get container status \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": rpc error: code = NotFound desc = could not find container \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": container with ID starting with e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e not found: ID does not exist"
Jul 07 11:42:48.125644 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.125621    2572 scope.go:117] "RemoveContainer" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"
Jul 07 11:42:48.125824 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.125800    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"} err="failed to get container status \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": rpc error: code = NotFound desc = could not find container \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": container with ID starting with 09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6 not found: ID does not exist"
Jul 07 11:42:48.125824 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.125824    2572 scope.go:117] "RemoveContainer" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"
Jul 07 11:42:48.126153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.126128    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"} err="failed to get container status \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": rpc error: code = NotFound desc = could not find container \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": container with ID starting with d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5 not found: ID does not exist"
Jul 07 11:42:48.126253 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.126156    2572 scope.go:117] "RemoveContainer" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"
Jul 07 11:42:48.126647 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.126623    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"} err="failed to get container status \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": rpc error: code = NotFound desc = could not find container \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": container with ID starting with c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665 not found: ID does not exist"
Jul 07 11:42:48.126749 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.126649    2572 scope.go:117] "RemoveContainer" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"
Jul 07 11:42:48.126939 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.126921    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"} err="failed to get container status \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": rpc error: code = NotFound desc = could not find container \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": container with ID starting with a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6 not found: ID does not exist"
Jul 07 11:42:48.126988 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.126939    2572 scope.go:117] "RemoveContainer" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"
Jul 07 11:42:48.127166 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127151    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"} err="failed to get container status \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": rpc error: code = NotFound desc = could not find container \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": container with ID starting with 0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7 not found: ID does not exist"
Jul 07 11:42:48.127208 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127166    2572 scope.go:117] "RemoveContainer" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"
Jul 07 11:42:48.127366 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127351    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"} err="failed to get container status \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": rpc error: code = NotFound desc = could not find container \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": container with ID starting with 209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06 not found: ID does not exist"
Jul 07 11:42:48.127366 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127366    2572 scope.go:117] "RemoveContainer" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"
Jul 07 11:42:48.127617 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127597    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"} err="failed to get container status \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": rpc error: code = NotFound desc = could not find container \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": container with ID starting with e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e not found: ID does not exist"
Jul 07 11:42:48.127617 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127616    2572 scope.go:117] "RemoveContainer" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"
Jul 07 11:42:48.127855 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127839    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"} err="failed to get container status \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": rpc error: code = NotFound desc = could not find container \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": container with ID starting with 09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6 not found: ID does not exist"
Jul 07 11:42:48.127855 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127855    2572 scope.go:117] "RemoveContainer" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"
Jul 07 11:42:48.127932 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.127891    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.128064 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128049    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"} err="failed to get container status \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": rpc error: code = NotFound desc = could not find container \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": container with ID starting with d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5 not found: ID does not exist"
Jul 07 11:42:48.128105 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128064    2572 scope.go:117] "RemoveContainer" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"
Jul 07 11:42:48.128152 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128139    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="prometheus"
Jul 07 11:42:48.128196 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128155    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="prometheus"
Jul 07 11:42:48.128196 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128168    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="init-config-reloader"
Jul 07 11:42:48.128196 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128177    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="init-config-reloader"
Jul 07 11:42:48.128196 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128194    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy-thanos"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128200    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy-thanos"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128206    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy-web"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128211    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy-web"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128220    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="config-reloader"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128228    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="config-reloader"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128237    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128243    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128240    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"} err="failed to get container status \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": rpc error: code = NotFound desc = could not find container \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": container with ID starting with c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665 not found: ID does not exist"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128250    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="thanos-sidecar"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128254    2572 scope.go:117] "RemoveContainer" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"
Jul 07 11:42:48.128325 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128255    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="thanos-sidecar"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128353    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy-web"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128366    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="config-reloader"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128372    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy-thanos"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128378    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="prometheus"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128383    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="thanos-sidecar"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128390    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" containerName="kube-rbac-proxy"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128514    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"} err="failed to get container status \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": rpc error: code = NotFound desc = could not find container \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": container with ID starting with a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6 not found: ID does not exist"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128527    2572 scope.go:117] "RemoveContainer" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"
Jul 07 11:42:48.128730 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128719    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"} err="failed to get container status \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": rpc error: code = NotFound desc = could not find container \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": container with ID starting with 0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7 not found: ID does not exist"
Jul 07 11:42:48.129004 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128736    2572 scope.go:117] "RemoveContainer" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"
Jul 07 11:42:48.129004 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128960    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"} err="failed to get container status \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": rpc error: code = NotFound desc = could not find container \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": container with ID starting with 209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06 not found: ID does not exist"
Jul 07 11:42:48.129004 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.128982    2572 scope.go:117] "RemoveContainer" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"
Jul 07 11:42:48.129221 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.129203    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"} err="failed to get container status \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": rpc error: code = NotFound desc = could not find container \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": container with ID starting with e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e not found: ID does not exist"
Jul 07 11:42:48.129273 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.129221    2572 scope.go:117] "RemoveContainer" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"
Jul 07 11:42:48.129419 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.129401    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"} err="failed to get container status \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": rpc error: code = NotFound desc = could not find container \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": container with ID starting with 09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6 not found: ID does not exist"
Jul 07 11:42:48.129459 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.129420    2572 scope.go:117] "RemoveContainer" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"
Jul 07 11:42:48.129611 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.129597    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"} err="failed to get container status \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": rpc error: code = NotFound desc = could not find container \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": container with ID starting with d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5 not found: ID does not exist"
Jul 07 11:42:48.129652 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.129612    2572 scope.go:117] "RemoveContainer" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"
Jul 07 11:42:48.129826 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.129807    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"} err="failed to get container status \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": rpc error: code = NotFound desc = could not find container \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": container with ID starting with c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665 not found: ID does not exist"
Jul 07 11:42:48.129879 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.129826    2572 scope.go:117] "RemoveContainer" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"
Jul 07 11:42:48.130041 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130023    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"} err="failed to get container status \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": rpc error: code = NotFound desc = could not find container \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": container with ID starting with a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6 not found: ID does not exist"
Jul 07 11:42:48.130101 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130044    2572 scope.go:117] "RemoveContainer" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"
Jul 07 11:42:48.130250 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130232    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"} err="failed to get container status \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": rpc error: code = NotFound desc = could not find container \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": container with ID starting with 0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7 not found: ID does not exist"
Jul 07 11:42:48.130296 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130252    2572 scope.go:117] "RemoveContainer" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"
Jul 07 11:42:48.130456 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130440    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"} err="failed to get container status \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": rpc error: code = NotFound desc = could not find container \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": container with ID starting with 209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06 not found: ID does not exist"
Jul 07 11:42:48.130506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130457    2572 scope.go:117] "RemoveContainer" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"
Jul 07 11:42:48.130645 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130627    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"} err="failed to get container status \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": rpc error: code = NotFound desc = could not find container \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": container with ID starting with e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e not found: ID does not exist"
Jul 07 11:42:48.130705 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130646    2572 scope.go:117] "RemoveContainer" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"
Jul 07 11:42:48.130854 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130837    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"} err="failed to get container status \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": rpc error: code = NotFound desc = could not find container \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": container with ID starting with 09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6 not found: ID does not exist"
Jul 07 11:42:48.130900 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.130855    2572 scope.go:117] "RemoveContainer" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"
Jul 07 11:42:48.131070 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131051    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"} err="failed to get container status \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": rpc error: code = NotFound desc = could not find container \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": container with ID starting with d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5 not found: ID does not exist"
Jul 07 11:42:48.131140 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131071    2572 scope.go:117] "RemoveContainer" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"
Jul 07 11:42:48.131305 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131286    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"} err="failed to get container status \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": rpc error: code = NotFound desc = could not find container \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": container with ID starting with c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665 not found: ID does not exist"
Jul 07 11:42:48.131352 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131306    2572 scope.go:117] "RemoveContainer" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"
Jul 07 11:42:48.131496 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131479    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"} err="failed to get container status \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": rpc error: code = NotFound desc = could not find container \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": container with ID starting with a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6 not found: ID does not exist"
Jul 07 11:42:48.131535 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131497    2572 scope.go:117] "RemoveContainer" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"
Jul 07 11:42:48.131725 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131708    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"} err="failed to get container status \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": rpc error: code = NotFound desc = could not find container \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": container with ID starting with 0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7 not found: ID does not exist"
Jul 07 11:42:48.131725 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131724    2572 scope.go:117] "RemoveContainer" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"
Jul 07 11:42:48.131914 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131897    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"} err="failed to get container status \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": rpc error: code = NotFound desc = could not find container \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": container with ID starting with 209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06 not found: ID does not exist"
Jul 07 11:42:48.131957 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131916    2572 scope.go:117] "RemoveContainer" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"
Jul 07 11:42:48.131957 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.131908    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.132363 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.132144    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"} err="failed to get container status \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": rpc error: code = NotFound desc = could not find container \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": container with ID starting with e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e not found: ID does not exist"
Jul 07 11:42:48.132363 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.132169    2572 scope.go:117] "RemoveContainer" containerID="09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"
Jul 07 11:42:48.132637 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.132616    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6"} err="failed to get container status \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": rpc error: code = NotFound desc = could not find container \"09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6\": container with ID starting with 09b52a87527201a32c0ff1d7614ece1f09744e2dc122786d4ace47e22c7f86b6 not found: ID does not exist"
Jul 07 11:42:48.132720 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.132638    2572 scope.go:117] "RemoveContainer" containerID="d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"
Jul 07 11:42:48.132885 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.132865    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5"} err="failed to get container status \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": rpc error: code = NotFound desc = could not find container \"d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5\": container with ID starting with d9c8ec91e271fcaedf94e57495ce68a4a8ed3743e0087c906eee2126dbae0fe5 not found: ID does not exist"
Jul 07 11:42:48.132959 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.132887    2572 scope.go:117] "RemoveContainer" containerID="c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"
Jul 07 11:42:48.133152 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.133135    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665"} err="failed to get container status \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": rpc error: code = NotFound desc = could not find container \"c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665\": container with ID starting with c9894f83315ccdcb8e68b45822f61419ffcfcbffdbfc55b59a60eb33666f0665 not found: ID does not exist"
Jul 07 11:42:48.133200 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.133153    2572 scope.go:117] "RemoveContainer" containerID="a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"
Jul 07 11:42:48.133378 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.133357    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6"} err="failed to get container status \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": rpc error: code = NotFound desc = could not find container \"a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6\": container with ID starting with a1a5f5ea641b5def1858349ceec223fd946a127408d7bb4ca474899e02e560e6 not found: ID does not exist"
Jul 07 11:42:48.133450 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.133378    2572 scope.go:117] "RemoveContainer" containerID="0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"
Jul 07 11:42:48.133729 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.133680    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7"} err="failed to get container status \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": rpc error: code = NotFound desc = could not find container \"0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7\": container with ID starting with 0a1982b4327264d158f19be86f6fa3837b6712b69b3da12cd7faebfaf262b4b7 not found: ID does not exist"
Jul 07 11:42:48.133729 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.133707    2572 scope.go:117] "RemoveContainer" containerID="209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"
Jul 07 11:42:48.134292 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.134037    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06"} err="failed to get container status \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": rpc error: code = NotFound desc = could not find container \"209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06\": container with ID starting with 209e71f3b863f63b3d286e63eec7b2fa7782cf4e31a344b76a90a9155dd35c06 not found: ID does not exist"
Jul 07 11:42:48.134292 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.134059    2572 scope.go:117] "RemoveContainer" containerID="e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"
Jul 07 11:42:48.134420 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.134363    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e"} err="failed to get container status \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": rpc error: code = NotFound desc = could not find container \"e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e\": container with ID starting with e89172fbcfae472246ef9e1d4fc64b8810b7f874b7adefc3bff7c6e34201c44e not found: ID does not exist"
Jul 07 11:42:48.134773 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.134729    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 11:42:48.134773 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.134764    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 11:42:48.135019 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135004    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 11:42:48.135087 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135075    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9e2oirm9slk82\""
Jul 07 11:42:48.135146 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135104    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 11:42:48.135217 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135200    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 11:42:48.135340 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135323    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 11:42:48.135395 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135359    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zkm6m\""
Jul 07 11:42:48.135444 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135414    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 11:42:48.135496 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135469    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 11:42:48.135546 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135498    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 11:42:48.135596 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135558    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 11:42:48.135828 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135810    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 11:42:48.135902 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135887    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 11:42:48.135954 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.135890    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 11:42:48.141161 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.141143    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 11:42:48.142894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.142877    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 11:42:48.146480 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.146462    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.156525 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156505    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156612 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156532    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156612 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156549    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh4rj\" (UniqueName: \"kubernetes.io/projected/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-kube-api-access-zh4rj\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156612 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156572    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156853 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156633    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-web-config\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156853 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156714    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156853 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156853 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156773    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156853 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-config-out\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.156853 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156831    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156877    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156902    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156930    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.156995    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157085    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157132 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157111    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-config\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157221    2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-config-out\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157236    2572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157252    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-db\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157266    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157281    2572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-metrics-client-certs\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157295    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157309    2572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157324    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157337    2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-web-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157352    2572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-grpc-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157364    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-configmap-metrics-client-ca\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157378    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157394    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157408    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncbgr\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-kube-api-access-ncbgr\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157421    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.157506 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157435    2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cff454b5-56b8-42d1-b8fe-e56c355e4519-tls-assets\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.158024 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157449    2572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-kube-rbac-proxy\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.158024 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157463    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cff454b5-56b8-42d1-b8fe-e56c355e4519-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.158024 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.157476    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cff454b5-56b8-42d1-b8fe-e56c355e4519-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.258577 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258560    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258698 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258584    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258698 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258698 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258622    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258698 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258637    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-config\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258698 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258653    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258827    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258879    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh4rj\" (UniqueName: \"kubernetes.io/projected/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-kube-api-access-zh4rj\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258907    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.258944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258931    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-web-config\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258960    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.258990    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259041    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-config-out\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259118    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259206 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259592 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259345    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259592 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259405    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259592 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259438    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259761 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259700    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.259946 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.259910    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.260350 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.260318    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.261616 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.261587    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.261721 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.261615    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.262085 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.261834    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-config-out\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.262170 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.262152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.262336 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.262278    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-config\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.262972 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.262945    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.263069 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.263019    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.263374 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.263351    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.263514 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.263496    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.264054 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.264032    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.264128 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.264103    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-web-config\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.264210 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.264194    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.264972 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.264953    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.269840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.269821    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh4rj\" (UniqueName: \"kubernetes.io/projected/17fc6f49-0c45-4c5e-8613-c6e4c3da58fa-kube-api-access-zh4rj\") pod \"prometheus-k8s-0\" (UID: \"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.438184 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.438152    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cff454b5-56b8-42d1-b8fe-e56c355e4519" path="/var/lib/kubelet/pods/cff454b5-56b8-42d1-b8fe-e56c355e4519/volumes"
Jul 07 11:42:48.443418 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.443402    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.574565 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:48.574507    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.577074 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:42:48.577050    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fc6f49_0c45_4c5e_8613_c6e4c3da58fa.slice/crio-2faf4377455d5dcfc14bd6982eeb0428023c7b2db3048e1b57bb00af391099cb WatchSource:0}: Error finding container 2faf4377455d5dcfc14bd6982eeb0428023c7b2db3048e1b57bb00af391099cb: Status 404 returned error can't find the container with id 2faf4377455d5dcfc14bd6982eeb0428023c7b2db3048e1b57bb00af391099cb
Jul 07 11:42:49.080092 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:49.080060    2572 generic.go:356] "Generic (PLEG): container finished" podID="17fc6f49-0c45-4c5e-8613-c6e4c3da58fa" containerID="855b39475024fe3ab789f1d10803417f243357598787771b0d99e445be09e7b9" exitCode=0
Jul 07 11:42:49.080428 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:49.080117    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa","Type":"ContainerDied","Data":"855b39475024fe3ab789f1d10803417f243357598787771b0d99e445be09e7b9"}
Jul 07 11:42:49.080428 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:49.080170    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa","Type":"ContainerStarted","Data":"2faf4377455d5dcfc14bd6982eeb0428023c7b2db3048e1b57bb00af391099cb"}
Jul 07 11:42:50.087435 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.087398    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa","Type":"ContainerStarted","Data":"1538761173c7dcec27ae8568d95e1cfda422015f5fec98d9c2fc5e6de76a6694"}
Jul 07 11:42:50.087824 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.087477    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa","Type":"ContainerStarted","Data":"da87418738f7be45c58709dcf01797b98da3cb231c85e3383f88d91f9099524a"}
Jul 07 11:42:50.087824 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.087516    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa","Type":"ContainerStarted","Data":"a8e9c0f0e0e3ae715016c2641b94bc798dbf43f7e0f739f631ca0c5f72e8558e"}
Jul 07 11:42:50.087824 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.087556    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa","Type":"ContainerStarted","Data":"dfc6f5ff0faaf4d6b0fc6a95a32493d4fc3733c039089aea603c04c2ede26cd9"}
Jul 07 11:42:50.087824 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.087596    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa","Type":"ContainerStarted","Data":"7a7108078820aefe145ab570be3db4600ae18696213ba5414640c429caa13264"}
Jul 07 11:42:50.087824 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.087631    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17fc6f49-0c45-4c5e-8613-c6e4c3da58fa","Type":"ContainerStarted","Data":"7451d4bdcc48b5a689fb2da1d7f523a04a8796c5e55be4a75d7a155bf6a31cd0"}
Jul 07 11:42:50.115888 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.115832    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.099123567 podStartE2EDuration="2.115811806s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.688239ms" isStatefulPod=false podCreationTimestamp="2026-07-07 11:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:42:50.114409369 +0000 UTC m=+156.309986390" watchObservedRunningTime="2026-07-07 11:42:50.115811806 +0000 UTC m=+156.311388828"
Jul 07 11:42:50.709652 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.709609    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:50.805976 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.805942    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:50.866038 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:50.866007    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:51.403372 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:51.403335    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:51.468990 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:51.468960    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:42:53.444483 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:42:53.444453    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:43:02.079676 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:02.079612    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:43:02.180910 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:02.180881    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:43:02.259976 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:02.259949    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:43:04.345676 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:04.345627    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:43:40.815623 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:40.815577    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:43:40.890130 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:40.890098    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:43:48.444384 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:48.444352    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:43:48.458953 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:48.458931    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:43:49.313219 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:49.313191    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:43:57.580354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:57.580316    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:43:57.674918 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:43:57.674895    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:01.731937 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:01.731899    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:01.832915 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:01.832887    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:01.932286 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:01.932254    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:02.468461 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:02.468425    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:02.575286 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:02.575254    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:09.515821 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:09.515764    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:09.572952 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:09.572919    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:12.323601 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:12.323559    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:12.423162 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:12.423130    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:12.480302 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:12.480272    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:20.363022 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:20.362979    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:20.464806 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:20.464769    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:44:20.535205 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:44:20.535170    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:45:14.209153 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:45:14.209125    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:45:14.212450 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:45:14.212423    2572 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 11:45:14.213193 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:45:14.213174    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:45:30.380273 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:45:30.380238    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:45:31.949424 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:45:31.949386    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:45:58.415944 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:45:58.415905    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:45:58.502224 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:45:58.502196    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:03.480613 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:03.480584    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:03.716389 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:03.716357    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:04.336248 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:04.336211    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:04.463320 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:04.463283    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:04.593405 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:04.593320    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:04.684638 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:04.684603    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:04.767058 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:04.767024    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:05.006750 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:05.006718    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:05.101020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:05.100987    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:05.206987 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:05.206957    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:05.408167 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:05.408083    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:46:05.508869 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:46:05.508839    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:22.979591 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:22.979555    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:23.146142 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:23.146111    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:23.334830 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:23.334800    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:23.578637 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:23.578607    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:23.745173 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:23.745099    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:23.979193 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:23.979156    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:31.969354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:31.969315    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:32.183239 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:32.183201    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:53.284630 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:53.284545    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:53.389023 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:53.388996    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:47:57.384290 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:47:57.384253    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:04.393700 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:04.393644    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:04.591035 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:04.591007    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:06.454116 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:06.454080    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:06.535242 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:06.535212    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:08.217174 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:08.217140    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:35.159065 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.159033    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:35.205935 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.205906    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zpxjl"]
Jul 07 11:48:35.208955 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.208939    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:35.212269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.212250    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 11:48:35.212521 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.212505    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-qt6qc\""
Jul 07 11:48:35.213275 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.213258    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 11:48:35.213597 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.213577    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 11:48:35.223281 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.223259    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zpxjl"]
Jul 07 11:48:35.259523 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.259501    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:35.293218 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.293200    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4d2ee5-014d-479f-b760-51fe6624f419-cert\") pod \"odh-model-controller-84b54b7f97-zpxjl\" (UID: \"7a4d2ee5-014d-479f-b760-51fe6624f419\") " pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:35.293302 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.293234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpjt\" (UniqueName: \"kubernetes.io/projected/7a4d2ee5-014d-479f-b760-51fe6624f419-kube-api-access-ljpjt\") pod \"odh-model-controller-84b54b7f97-zpxjl\" (UID: \"7a4d2ee5-014d-479f-b760-51fe6624f419\") " pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:35.366820 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.366797    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:35.394023 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.394003    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4d2ee5-014d-479f-b760-51fe6624f419-cert\") pod \"odh-model-controller-84b54b7f97-zpxjl\" (UID: \"7a4d2ee5-014d-479f-b760-51fe6624f419\") " pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:35.394099 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.394039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpjt\" (UniqueName: \"kubernetes.io/projected/7a4d2ee5-014d-479f-b760-51fe6624f419-kube-api-access-ljpjt\") pod \"odh-model-controller-84b54b7f97-zpxjl\" (UID: \"7a4d2ee5-014d-479f-b760-51fe6624f419\") " pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:35.396349 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.396332    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4d2ee5-014d-479f-b760-51fe6624f419-cert\") pod \"odh-model-controller-84b54b7f97-zpxjl\" (UID: \"7a4d2ee5-014d-479f-b760-51fe6624f419\") " pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:35.404612 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.404590    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpjt\" (UniqueName: \"kubernetes.io/projected/7a4d2ee5-014d-479f-b760-51fe6624f419-kube-api-access-ljpjt\") pod \"odh-model-controller-84b54b7f97-zpxjl\" (UID: \"7a4d2ee5-014d-479f-b760-51fe6624f419\") " pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:35.519168 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.519149    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:35.679383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.679352    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zpxjl"]
Jul 07 11:48:35.682628 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:48:35.682601    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4d2ee5_014d_479f_b760_51fe6624f419.slice/crio-adaaa06345918a513139f0ef525ced6ec4a80d2c2b9aa4eb1601b5c2d5de788f WatchSource:0}: Error finding container adaaa06345918a513139f0ef525ced6ec4a80d2c2b9aa4eb1601b5c2d5de788f: Status 404 returned error can't find the container with id adaaa06345918a513139f0ef525ced6ec4a80d2c2b9aa4eb1601b5c2d5de788f
Jul 07 11:48:35.684112 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.684095    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:48:35.766733 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:35.766696    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:36.060274 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:36.060243    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:36.130035 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:36.129978    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:36.313341 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:36.313269    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zpxjl" event={"ID":"7a4d2ee5-014d-479f-b760-51fe6624f419","Type":"ContainerStarted","Data":"adaaa06345918a513139f0ef525ced6ec4a80d2c2b9aa4eb1601b5c2d5de788f"}
Jul 07 11:48:39.327916 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:39.327885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zpxjl" event={"ID":"7a4d2ee5-014d-479f-b760-51fe6624f419","Type":"ContainerStarted","Data":"7f6a484f6b7b1c9ffa9aad11a484e2db7b99a1994c49c49bb98412cf30ecdea5"}
Jul 07 11:48:39.328295 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:39.327998    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:39.344428 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:39.344382    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-zpxjl" podStartSLOduration=1.351769485 podStartE2EDuration="4.344369611s" totalImagesPullingTime="2.992600126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:48:39.343184967 +0000 UTC m=+505.538761990" watchObservedRunningTime="2026-07-07 11:48:39.344369611 +0000 UTC m=+505.539946632"
Jul 07 11:48:50.333313 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:50.333278    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-zpxjl"
Jul 07 11:48:51.166455 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.166421    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-lhqjf"]
Jul 07 11:48:51.169820 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.169804    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lhqjf"
Jul 07 11:48:51.172278 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.172260    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2smsb\""
Jul 07 11:48:51.172278 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.172271    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 11:48:51.178137 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.178104    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lhqjf"]
Jul 07 11:48:51.310484 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.310461    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e920a33f-a9cd-4fd7-bb62-3507041a218c-model-cache\") pod \"s3-init-lhqjf\" (UID: \"e920a33f-a9cd-4fd7-bb62-3507041a218c\") " pod="kserve/s3-init-lhqjf"
Jul 07 11:48:51.310577 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.310488    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8hc\" (UniqueName: \"kubernetes.io/projected/e920a33f-a9cd-4fd7-bb62-3507041a218c-kube-api-access-wm8hc\") pod \"s3-init-lhqjf\" (UID: \"e920a33f-a9cd-4fd7-bb62-3507041a218c\") " pod="kserve/s3-init-lhqjf"
Jul 07 11:48:51.411246 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.411223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e920a33f-a9cd-4fd7-bb62-3507041a218c-model-cache\") pod \"s3-init-lhqjf\" (UID: \"e920a33f-a9cd-4fd7-bb62-3507041a218c\") " pod="kserve/s3-init-lhqjf"
Jul 07 11:48:51.411530 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.411250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8hc\" (UniqueName: \"kubernetes.io/projected/e920a33f-a9cd-4fd7-bb62-3507041a218c-kube-api-access-wm8hc\") pod \"s3-init-lhqjf\" (UID: \"e920a33f-a9cd-4fd7-bb62-3507041a218c\") " pod="kserve/s3-init-lhqjf"
Jul 07 11:48:51.411575 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.411546    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e920a33f-a9cd-4fd7-bb62-3507041a218c-model-cache\") pod \"s3-init-lhqjf\" (UID: \"e920a33f-a9cd-4fd7-bb62-3507041a218c\") " pod="kserve/s3-init-lhqjf"
Jul 07 11:48:51.420677 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.420605    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8hc\" (UniqueName: \"kubernetes.io/projected/e920a33f-a9cd-4fd7-bb62-3507041a218c-kube-api-access-wm8hc\") pod \"s3-init-lhqjf\" (UID: \"e920a33f-a9cd-4fd7-bb62-3507041a218c\") " pod="kserve/s3-init-lhqjf"
Jul 07 11:48:51.488047 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.488027    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lhqjf"
Jul 07 11:48:51.609173 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:51.609144    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lhqjf"]
Jul 07 11:48:51.612270 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:48:51.612248    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode920a33f_a9cd_4fd7_bb62_3507041a218c.slice/crio-692c7d490e71d9c39fd03d5563fc0973dc18f443a494e382009082f1d623eaf0 WatchSource:0}: Error finding container 692c7d490e71d9c39fd03d5563fc0973dc18f443a494e382009082f1d623eaf0: Status 404 returned error can't find the container with id 692c7d490e71d9c39fd03d5563fc0973dc18f443a494e382009082f1d623eaf0
Jul 07 11:48:52.377147 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:52.377100    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lhqjf" event={"ID":"e920a33f-a9cd-4fd7-bb62-3507041a218c","Type":"ContainerStarted","Data":"692c7d490e71d9c39fd03d5563fc0973dc18f443a494e382009082f1d623eaf0"}
Jul 07 11:48:55.395835 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:55.395796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lhqjf" event={"ID":"e920a33f-a9cd-4fd7-bb62-3507041a218c","Type":"ContainerStarted","Data":"862ed015801c0072eaa425677fef67c5b5afe017008daf7c28e6feb47edc5bf4"}
Jul 07 11:48:55.867205 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:55.867139    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:55.926414 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:55.926372    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:48:57.405437 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:57.405406    2572 generic.go:356] "Generic (PLEG): container finished" podID="e920a33f-a9cd-4fd7-bb62-3507041a218c" containerID="862ed015801c0072eaa425677fef67c5b5afe017008daf7c28e6feb47edc5bf4" exitCode=0
Jul 07 11:48:57.405835 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:48:57.405480    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lhqjf" event={"ID":"e920a33f-a9cd-4fd7-bb62-3507041a218c","Type":"ContainerDied","Data":"862ed015801c0072eaa425677fef67c5b5afe017008daf7c28e6feb47edc5bf4"}
Jul 07 11:49:02.431583 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:49:02.431553    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lhqjf" event={"ID":"e920a33f-a9cd-4fd7-bb62-3507041a218c","Type":"ContainerStarted","Data":"0929648e1598b9c3c58101762c35430624cc140bed818978ccc0072749379f0f"}
Jul 07 11:49:02.448102 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:49:02.448051    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-lhqjf" podStartSLOduration=2.153287262 podStartE2EDuration="11.448040244s" totalImagesPullingTime="7.582412485s" totalInitContainerRuntime="1.712340497s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:48:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:49:02.447180779 +0000 UTC m=+528.642757804" watchObservedRunningTime="2026-07-07 11:49:02.448040244 +0000 UTC m=+528.643617266"
Jul 07 11:50:07.658726 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:07.658692    2572 generic.go:356] "Generic (PLEG): container finished" podID="e920a33f-a9cd-4fd7-bb62-3507041a218c" containerID="0929648e1598b9c3c58101762c35430624cc140bed818978ccc0072749379f0f" exitCode=0
Jul 07 11:50:07.659124 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:07.658766    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lhqjf" event={"ID":"e920a33f-a9cd-4fd7-bb62-3507041a218c","Type":"ContainerDied","Data":"0929648e1598b9c3c58101762c35430624cc140bed818978ccc0072749379f0f"}
Jul 07 11:50:08.786091 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:08.786070    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lhqjf"
Jul 07 11:50:08.861199 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:08.861173    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e920a33f-a9cd-4fd7-bb62-3507041a218c-model-cache\") pod \"e920a33f-a9cd-4fd7-bb62-3507041a218c\" (UID: \"e920a33f-a9cd-4fd7-bb62-3507041a218c\") "
Jul 07 11:50:08.861330 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:08.861208    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm8hc\" (UniqueName: \"kubernetes.io/projected/e920a33f-a9cd-4fd7-bb62-3507041a218c-kube-api-access-wm8hc\") pod \"e920a33f-a9cd-4fd7-bb62-3507041a218c\" (UID: \"e920a33f-a9cd-4fd7-bb62-3507041a218c\") "
Jul 07 11:50:08.863259 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:08.863234    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e920a33f-a9cd-4fd7-bb62-3507041a218c-kube-api-access-wm8hc" (OuterVolumeSpecName: "kube-api-access-wm8hc") pod "e920a33f-a9cd-4fd7-bb62-3507041a218c" (UID: "e920a33f-a9cd-4fd7-bb62-3507041a218c"). InnerVolumeSpecName "kube-api-access-wm8hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:50:08.880527 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:08.880502    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e920a33f-a9cd-4fd7-bb62-3507041a218c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e920a33f-a9cd-4fd7-bb62-3507041a218c" (UID: "e920a33f-a9cd-4fd7-bb62-3507041a218c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:50:08.962247 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:08.962192    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e920a33f-a9cd-4fd7-bb62-3507041a218c-model-cache\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:50:08.962247 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:08.962212    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm8hc\" (UniqueName: \"kubernetes.io/projected/e920a33f-a9cd-4fd7-bb62-3507041a218c-kube-api-access-wm8hc\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:50:09.667088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:09.667054    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lhqjf" event={"ID":"e920a33f-a9cd-4fd7-bb62-3507041a218c","Type":"ContainerDied","Data":"692c7d490e71d9c39fd03d5563fc0973dc18f443a494e382009082f1d623eaf0"}
Jul 07 11:50:09.667088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:09.667091    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692c7d490e71d9c39fd03d5563fc0973dc18f443a494e382009082f1d623eaf0"
Jul 07 11:50:09.667088 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:09.667061    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lhqjf"
Jul 07 11:50:10.033137 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:10.033106    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:10.132890 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:10.132853    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:14.237102 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:14.237074    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:50:14.237544 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:14.237509    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:50:21.123983 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.123953    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:21.161703 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.161677    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"]
Jul 07 11:50:21.162104 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.162090    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e920a33f-a9cd-4fd7-bb62-3507041a218c" containerName="download-hf-model"
Jul 07 11:50:21.162147 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.162106    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e920a33f-a9cd-4fd7-bb62-3507041a218c" containerName="download-hf-model"
Jul 07 11:50:21.162147 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.162126    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e920a33f-a9cd-4fd7-bb62-3507041a218c" containerName="s3-init"
Jul 07 11:50:21.162147 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.162136    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e920a33f-a9cd-4fd7-bb62-3507041a218c" containerName="s3-init"
Jul 07 11:50:21.162240 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.162229    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e920a33f-a9cd-4fd7-bb62-3507041a218c" containerName="s3-init"
Jul 07 11:50:21.165754 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.165737    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.168161 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.168141    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7cc25-kube-rbac-proxy-sar-config\""
Jul 07 11:50:21.168265 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.168142    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7cc25-predictor-serving-cert\""
Jul 07 11:50:21.168411 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.168397    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 11:50:21.168998 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.168984    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pntcj\""
Jul 07 11:50:21.169164 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.169153    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 11:50:21.175360 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.175332    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"]
Jul 07 11:50:21.228494 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.228471    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:21.250296 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.250271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e594d-e47a-4d1d-b85c-a3334539ed05-proxy-tls\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.250385 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.250305    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlfb\" (UniqueName: \"kubernetes.io/projected/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-api-access-bdlfb\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.250385 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.250332    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.350997 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.350973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlfb\" (UniqueName: \"kubernetes.io/projected/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-api-access-bdlfb\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.351080 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.351013    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.351120 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.351089    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e594d-e47a-4d1d-b85c-a3334539ed05-proxy-tls\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.351706 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.351690    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.353497 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.353476    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e594d-e47a-4d1d-b85c-a3334539ed05-proxy-tls\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.359031 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.359012    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlfb\" (UniqueName: \"kubernetes.io/projected/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-api-access-bdlfb\") pod \"success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") " pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.415761 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.415694    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:21.458675 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.458632    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"]
Jul 07 11:50:21.462055 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.462040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.464547 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.464530    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 11:50:21.464943 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.464907    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 07 11:50:21.464943 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.464925    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 11:50:21.465101 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.464940    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 07 11:50:21.471998 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.471979    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"]
Jul 07 11:50:21.476989 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.476960    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:21.520310 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.520279    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:21.552420 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.552397    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.552534 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.552437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wspxr\" (UniqueName: \"kubernetes.io/projected/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-api-access-wspxr\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.552534 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.552466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.552644 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.552548    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d638385e-c8ef-4f64-a1f9-d34e94619741-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.552644 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.552594    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.612757 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.612684    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"]
Jul 07 11:50:21.643320 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.643274    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:21.653293 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.653266    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.653395 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.653325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.653474 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.653447    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspxr\" (UniqueName: \"kubernetes.io/projected/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-api-access-wspxr\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.653542 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.653516    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.653611 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.653561    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d638385e-c8ef-4f64-a1f9-d34e94619741-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.653695 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:50:21.653631    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 11:50:21.653747 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:50:21.653726    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls podName:d638385e-c8ef-4f64-a1f9-d34e94619741 nodeName:}" failed. No retries permitted until 2026-07-07 11:50:22.153705248 +0000 UTC m=+608.349282263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls") pod "isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" (UID: "d638385e-c8ef-4f64-a1f9-d34e94619741") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 11:50:21.653956 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.653931    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d638385e-c8ef-4f64-a1f9-d34e94619741-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.654155 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.654138    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.654216 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.654176    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.668294 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.668240    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspxr\" (UniqueName: \"kubernetes.io/projected/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-api-access-wspxr\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:21.676429 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.676404    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"]
Jul 07 11:50:21.680885 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.680871    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.683314 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.683298    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 07 11:50:21.683401 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.683339    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 07 11:50:21.690324 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.690298    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"]
Jul 07 11:50:21.710497 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.710475    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" event={"ID":"412e594d-e47a-4d1d-b85c-a3334539ed05","Type":"ContainerStarted","Data":"559592dc48fcb0f48f4e5bfcbccaae2ac64fe705cb938db82a7d7fe5a40a3d37"}
Jul 07 11:50:21.717115 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.717087    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:21.754249 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.754222    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh444\" (UniqueName: \"kubernetes.io/projected/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-api-access-zh444\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.754340 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.754264    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.754340 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.754293    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.754340 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.754338    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.754459 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.754380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.771042 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.771021    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"]
Jul 07 11:50:21.774363 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.774348    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.777269 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.777246    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 07 11:50:21.777345 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.777249    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 07 11:50:21.787605 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.787587    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"]
Jul 07 11:50:21.799962 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.799939    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:21.855165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855145    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.855247 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.855247 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855195    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rns7\" (UniqueName: \"kubernetes.io/projected/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-api-access-2rns7\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.855319 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh444\" (UniqueName: \"kubernetes.io/projected/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-api-access-zh444\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.855319 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.855391 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855334    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.855391 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.855472 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.855472 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855435    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.855551 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855508    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.855919 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.855898    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.856110 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.856094    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.856183 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.856094    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.857563 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.857542    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.866331 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.866314    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh444\" (UniqueName: \"kubernetes.io/projected/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-api-access-zh444\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4vm2\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:21.905098 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.905073    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:21.956305 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.956257    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.956305 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.956282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rns7\" (UniqueName: \"kubernetes.io/projected/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-api-access-2rns7\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.956445 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.956312    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.956445 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.956334    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.956445 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.956352    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.956595 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:50:21.956448    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 07 11:50:21.956595 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:50:21.956520    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls podName:e19a862e-90b6-4fb6-aa63-6f430ce4652e nodeName:}" failed. No retries permitted until 2026-07-07 11:50:22.456500106 +0000 UTC m=+608.652077112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls") pod "isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" (UID: "e19a862e-90b6-4fb6-aa63-6f430ce4652e") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 07 11:50:21.956757 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.956738    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.957002 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.956980    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.957054 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.957034    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.965642 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.965617    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rns7\" (UniqueName: \"kubernetes.io/projected/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-api-access-2rns7\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:21.991767 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:21.991739    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:22.043505 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.043472    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:22.118056 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.118035    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"]
Jul 07 11:50:22.119642 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:50:22.119610    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a85d9c_f4b1_44f7_a758_bfdaca67607b.slice/crio-71ab62e07ead73fefa98f145c38022deacac9a2346260138f9287bb8dd6d3a7d WatchSource:0}: Error finding container 71ab62e07ead73fefa98f145c38022deacac9a2346260138f9287bb8dd6d3a7d: Status 404 returned error can't find the container with id 71ab62e07ead73fefa98f145c38022deacac9a2346260138f9287bb8dd6d3a7d
Jul 07 11:50:22.158580 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.158554    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:22.158892 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:50:22.158700    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 11:50:22.158892 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:50:22.158753    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls podName:d638385e-c8ef-4f64-a1f9-d34e94619741 nodeName:}" failed. No retries permitted until 2026-07-07 11:50:23.15873618 +0000 UTC m=+609.354313180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls") pod "isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" (UID: "d638385e-c8ef-4f64-a1f9-d34e94619741") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 11:50:22.462705 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.462180    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:22.472450 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.472396    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:22.486023 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.485988    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:50:22.684167 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.684137    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:22.723277 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.723205    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" event={"ID":"e7a85d9c-f4b1-44f7-a758-bfdaca67607b","Type":"ContainerStarted","Data":"0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6"}
Jul 07 11:50:22.723434 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.723282    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" event={"ID":"e7a85d9c-f4b1-44f7-a758-bfdaca67607b","Type":"ContainerStarted","Data":"71ab62e07ead73fefa98f145c38022deacac9a2346260138f9287bb8dd6d3a7d"}
Jul 07 11:50:22.913403 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:22.913258    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"]
Jul 07 11:50:22.922816 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:50:22.922773    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19a862e_90b6_4fb6_aa63_6f430ce4652e.slice/crio-d2f42d8ab2ad18bad8d823b3f9c06bdd57e4f42697ab178b0cc7d0440619fa3a WatchSource:0}: Error finding container d2f42d8ab2ad18bad8d823b3f9c06bdd57e4f42697ab178b0cc7d0440619fa3a: Status 404 returned error can't find the container with id d2f42d8ab2ad18bad8d823b3f9c06bdd57e4f42697ab178b0cc7d0440619fa3a
Jul 07 11:50:23.172678 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:23.172342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:23.177003 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:23.176972    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-8b777f59f-2h82x\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:23.273579 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:23.273268    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:23.468312 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:23.467730    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"]
Jul 07 11:50:23.734195 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:23.734146    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" event={"ID":"d638385e-c8ef-4f64-a1f9-d34e94619741","Type":"ContainerStarted","Data":"4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274"}
Jul 07 11:50:23.734373 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:23.734217    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" event={"ID":"d638385e-c8ef-4f64-a1f9-d34e94619741","Type":"ContainerStarted","Data":"f07f905f0986ca648b5abe4267f365f673e8226c625189a074b6bf65ae26d221"}
Jul 07 11:50:23.738851 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:23.738703    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" event={"ID":"e19a862e-90b6-4fb6-aa63-6f430ce4652e","Type":"ContainerStarted","Data":"ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706"}
Jul 07 11:50:23.738851 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:23.738760    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" event={"ID":"e19a862e-90b6-4fb6-aa63-6f430ce4652e","Type":"ContainerStarted","Data":"d2f42d8ab2ad18bad8d823b3f9c06bdd57e4f42697ab178b0cc7d0440619fa3a"}
Jul 07 11:50:27.763151 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:27.763113    2572 generic.go:356] "Generic (PLEG): container finished" podID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerID="0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6" exitCode=0
Jul 07 11:50:27.763620 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:27.763190    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" event={"ID":"e7a85d9c-f4b1-44f7-a758-bfdaca67607b","Type":"ContainerDied","Data":"0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6"}
Jul 07 11:50:27.765794 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:27.765723    2572 generic.go:356] "Generic (PLEG): container finished" podID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerID="ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706" exitCode=0
Jul 07 11:50:27.765794 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:27.765758    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" event={"ID":"e19a862e-90b6-4fb6-aa63-6f430ce4652e","Type":"ContainerDied","Data":"ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706"}
Jul 07 11:50:31.788458 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:31.788419    2572 generic.go:356] "Generic (PLEG): container finished" podID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerID="4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274" exitCode=0
Jul 07 11:50:31.788994 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:31.788509    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" event={"ID":"d638385e-c8ef-4f64-a1f9-d34e94619741","Type":"ContainerDied","Data":"4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274"}
Jul 07 11:50:37.821178 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:37.821122    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" event={"ID":"412e594d-e47a-4d1d-b85c-a3334539ed05","Type":"ContainerStarted","Data":"e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58"}
Jul 07 11:50:41.843886 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.843811    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" event={"ID":"e19a862e-90b6-4fb6-aa63-6f430ce4652e","Type":"ContainerStarted","Data":"6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4"}
Jul 07 11:50:41.843886 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.843864    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" event={"ID":"e19a862e-90b6-4fb6-aa63-6f430ce4652e","Type":"ContainerStarted","Data":"ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454"}
Jul 07 11:50:41.844410 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.844388    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:41.844514 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.844454    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:41.846061 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.846017    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:50:41.846627 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.846600    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" event={"ID":"412e594d-e47a-4d1d-b85c-a3334539ed05","Type":"ContainerStarted","Data":"41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31"}
Jul 07 11:50:41.846783 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.846769    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:41.849410 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.849389    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" event={"ID":"d638385e-c8ef-4f64-a1f9-d34e94619741","Type":"ContainerStarted","Data":"65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b"}
Jul 07 11:50:41.849518 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.849438    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" event={"ID":"d638385e-c8ef-4f64-a1f9-d34e94619741","Type":"ContainerStarted","Data":"c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b"}
Jul 07 11:50:41.849651 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.849633    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:41.865500 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.865460    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podStartSLOduration=6.745133082 podStartE2EDuration="20.865448088s" totalImagesPullingTime="10.054074907s" totalInitContainerRuntime="4.066240099s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:50:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:50:41.864101973 +0000 UTC m=+628.059679021" watchObservedRunningTime="2026-07-07 11:50:41.865448088 +0000 UTC m=+628.061025110"
Jul 07 11:50:41.882565 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:41.882530    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podStartSLOduration=1.252805853 podStartE2EDuration="20.882518671s" totalImagesPullingTime="19.629712818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:50:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:50:41.881447809 +0000 UTC m=+628.077024825" watchObservedRunningTime="2026-07-07 11:50:41.882518671 +0000 UTC m=+628.078095692"
Jul 07 11:50:42.854759 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:42.854275    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:50:42.854759 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:42.854691    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:42.854759 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:42.854752    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:42.855812 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:42.855783    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:42.855930 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:42.855780    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:50:43.858501 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:43.858316    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:50:43.858501 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:43.858457    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:47.858913 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:47.858874    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:50:47.859498 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:47.859470    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:50:47.896376 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:47.896323    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podStartSLOduration=14.809540973 podStartE2EDuration="26.896307837s" totalImagesPullingTime="7.890059614s" totalInitContainerRuntime="4.19670725s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:50:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:50:41.907064346 +0000 UTC m=+628.102641369" watchObservedRunningTime="2026-07-07 11:50:47.896307837 +0000 UTC m=+634.091884860"
Jul 07 11:50:48.863229 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:48.863198    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:50:48.863716 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:48.863563    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:50:48.863788 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:48.863769    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:50:48.864262 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:48.864234    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:55.911645 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:55.911611    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" event={"ID":"e7a85d9c-f4b1-44f7-a758-bfdaca67607b","Type":"ContainerStarted","Data":"194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448"}
Jul 07 11:50:55.912001 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:55.911690    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" event={"ID":"e7a85d9c-f4b1-44f7-a758-bfdaca67607b","Type":"ContainerStarted","Data":"78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009"}
Jul 07 11:50:55.912001 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:55.911950    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:55.912096 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:55.912085    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:50:55.913012 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:55.912993    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:50:55.933288 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:55.933227    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podStartSLOduration=3.147830686 podStartE2EDuration="34.933207445s" totalImagesPullingTime="27.652884421s" totalInitContainerRuntime="4.132492338s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:50:55.930468587 +0000 UTC m=+642.126045626" watchObservedRunningTime="2026-07-07 11:50:55.933207445 +0000 UTC m=+642.128784471"
Jul 07 11:50:56.915703 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:56.915651    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:50:57.860387 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:57.860349    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:50:58.864740 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:58.864694    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:50:58.865180 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:50:58.864693    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:51:01.920017 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:01.919989    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:51:01.920541 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:01.920513    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:51:07.859763 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:07.859726    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:51:08.866472 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:08.866383    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:51:08.866887 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:08.866383    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:51:11.921395 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:11.921361    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:51:17.860183 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:17.860135    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:51:18.864532 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:18.864492    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:51:18.864532 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:18.864519    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:51:21.920836 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:21.920796    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:51:27.860255 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:27.860215    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:51:28.867571 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:28.867518    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:51:28.868023 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:28.867870    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:51:31.920535 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:31.920501    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:51:37.859957 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:37.859915    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:51:38.863793 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:38.863757    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:51:41.187256 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.187221    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"]
Jul 07 11:51:41.189466 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.189450    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:41.192911 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.192890    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7cc25-serving-cert\""
Jul 07 11:51:41.193018 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.192891    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7cc25-kube-rbac-proxy-sar-config\""
Jul 07 11:51:41.201592 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.201571    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"]
Jul 07 11:51:41.210072 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.210045    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:51:41.234951 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.234925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb373f2-d224-465e-8125-a85cef86ffb8-openshift-service-ca-bundle\") pod \"switch-graph-7cc25-6c975c9dbd-6v5mr\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") " pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:41.235076 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.235025    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls\") pod \"switch-graph-7cc25-6c975c9dbd-6v5mr\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") " pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:41.336375 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.336327    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls\") pod \"switch-graph-7cc25-6c975c9dbd-6v5mr\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") " pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:41.336550 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.336389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb373f2-d224-465e-8125-a85cef86ffb8-openshift-service-ca-bundle\") pod \"switch-graph-7cc25-6c975c9dbd-6v5mr\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") " pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:41.336550 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:51:41.336475    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-7cc25-serving-cert: secret "switch-graph-7cc25-serving-cert" not found
Jul 07 11:51:41.336550 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:51:41.336540    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls podName:3cb373f2-d224-465e-8125-a85cef86ffb8 nodeName:}" failed. No retries permitted until 2026-07-07 11:51:41.836521802 +0000 UTC m=+688.032098803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls") pod "switch-graph-7cc25-6c975c9dbd-6v5mr" (UID: "3cb373f2-d224-465e-8125-a85cef86ffb8") : secret "switch-graph-7cc25-serving-cert" not found
Jul 07 11:51:41.337222 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.337192    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb373f2-d224-465e-8125-a85cef86ffb8-openshift-service-ca-bundle\") pod \"switch-graph-7cc25-6c975c9dbd-6v5mr\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") " pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:41.840788 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.840749    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls\") pod \"switch-graph-7cc25-6c975c9dbd-6v5mr\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") " pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:41.843072 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.843049    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls\") pod \"switch-graph-7cc25-6c975c9dbd-6v5mr\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") " pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:41.921319 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:41.921273    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:51:42.099020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:42.098942    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:42.222358 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:42.222329    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"]
Jul 07 11:51:42.224966 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:51:42.224931    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb373f2_d224_465e_8125_a85cef86ffb8.slice/crio-6a24a5a8cbf2aa0c5bf4d9ab0eb3084b678e7e3fc9f53671a1db970bb5a271e1 WatchSource:0}: Error finding container 6a24a5a8cbf2aa0c5bf4d9ab0eb3084b678e7e3fc9f53671a1db970bb5a271e1: Status 404 returned error can't find the container with id 6a24a5a8cbf2aa0c5bf4d9ab0eb3084b678e7e3fc9f53671a1db970bb5a271e1
Jul 07 11:51:43.098270 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:43.098239    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" event={"ID":"3cb373f2-d224-465e-8125-a85cef86ffb8","Type":"ContainerStarted","Data":"6a24a5a8cbf2aa0c5bf4d9ab0eb3084b678e7e3fc9f53671a1db970bb5a271e1"}
Jul 07 11:51:43.125947 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:43.125908    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:51:45.108898 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:45.108859    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" event={"ID":"3cb373f2-d224-465e-8125-a85cef86ffb8","Type":"ContainerStarted","Data":"66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd"}
Jul 07 11:51:45.109322 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:45.108931    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:45.127035 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:45.126990    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" podStartSLOduration=1.757416222 podStartE2EDuration="4.12697581s" totalImagesPullingTime="2.369559588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:51:45.126013955 +0000 UTC m=+691.321590978" watchObservedRunningTime="2026-07-07 11:51:45.12697581 +0000 UTC m=+691.322552831"
Jul 07 11:51:47.859850 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:47.859812    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:51:48.863828 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:48.863794    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:51:51.118562 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.118532    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:51:51.318785 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.318754    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"]
Jul 07 11:51:51.319039 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.318985    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25" containerID="cri-o://66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd" gracePeriod=30
Jul 07 11:51:51.352485 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.352454    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:51:51.463746 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.463678    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"]
Jul 07 11:51:51.463904 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.463883    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container" containerID="cri-o://e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58" gracePeriod=30
Jul 07 11:51:51.463988 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.463925    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kube-rbac-proxy" containerID="cri-o://41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31" gracePeriod=30
Jul 07 11:51:51.480642 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.480618    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"]
Jul 07 11:51:51.481286 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.481261    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:51:51.483221 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.483203    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.485635 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.485602    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5e1d2-predictor-serving-cert\""
Jul 07 11:51:51.485796 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.485780    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5e1d2-kube-rbac-proxy-sar-config\""
Jul 07 11:51:51.499082 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.499058    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"]
Jul 07 11:51:51.516580 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.516560    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a35811-de8e-4483-b222-f574a9566720-proxy-tls\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.516700 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.516680    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftmrf\" (UniqueName: \"kubernetes.io/projected/34a35811-de8e-4483-b222-f574a9566720-kube-api-access-ftmrf\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.516760 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.516744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a35811-de8e-4483-b222-f574a9566720-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.586501 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.586471    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:51:51.617652 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.617628    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a35811-de8e-4483-b222-f574a9566720-proxy-tls\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.617761 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.617698    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftmrf\" (UniqueName: \"kubernetes.io/projected/34a35811-de8e-4483-b222-f574a9566720-kube-api-access-ftmrf\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.617761 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.617743    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a35811-de8e-4483-b222-f574a9566720-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.618311 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.618294    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a35811-de8e-4483-b222-f574a9566720-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.619906 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.619890    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a35811-de8e-4483-b222-f574a9566720-proxy-tls\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.634305 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.634249    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftmrf\" (UniqueName: \"kubernetes.io/projected/34a35811-de8e-4483-b222-f574a9566720-kube-api-access-ftmrf\") pod \"success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") " pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.678612 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.678578    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:51:51.758845 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.758807    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:51:51.794044 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.794017    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:51.857460 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.857418    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:51:51.920951 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.920923    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:51:51.921351 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:51.921332    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"]
Jul 07 11:51:51.923397 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:51:51.923376    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a35811_de8e_4483_b222_f574a9566720.slice/crio-e944917093c50685e2a45dc6c15626f96af9e6dc71e96d61556a8017c3919458 WatchSource:0}: Error finding container e944917093c50685e2a45dc6c15626f96af9e6dc71e96d61556a8017c3919458: Status 404 returned error can't find the container with id e944917093c50685e2a45dc6c15626f96af9e6dc71e96d61556a8017c3919458
Jul 07 11:51:52.142494 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:52.142456    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" event={"ID":"34a35811-de8e-4483-b222-f574a9566720","Type":"ContainerStarted","Data":"c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398"}
Jul 07 11:51:52.143084 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:52.142560    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:52.143084 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:52.142593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" event={"ID":"34a35811-de8e-4483-b222-f574a9566720","Type":"ContainerStarted","Data":"8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf"}
Jul 07 11:51:52.143084 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:52.142625    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" event={"ID":"34a35811-de8e-4483-b222-f574a9566720","Type":"ContainerStarted","Data":"e944917093c50685e2a45dc6c15626f96af9e6dc71e96d61556a8017c3919458"}
Jul 07 11:51:52.144509 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:52.144480    2572 generic.go:356] "Generic (PLEG): container finished" podID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerID="41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31" exitCode=2
Jul 07 11:51:52.144617 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:52.144517    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" event={"ID":"412e594d-e47a-4d1d-b85c-a3334539ed05","Type":"ContainerDied","Data":"41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31"}
Jul 07 11:51:52.166097 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:52.166046    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podStartSLOduration=1.166032223 podStartE2EDuration="1.166032223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:51:52.16480347 +0000 UTC m=+698.360380509" watchObservedRunningTime="2026-07-07 11:51:52.166032223 +0000 UTC m=+698.361609246"
Jul 07 11:51:53.148464 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:53.148431    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:53.149993 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:53.149961    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 11:51:53.859520 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:53.859468    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 07 11:51:54.152894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.152801    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 11:51:54.408034 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.407981    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:51:54.442591 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.442563    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-rbac-proxy-sar-config\") pod \"412e594d-e47a-4d1d-b85c-a3334539ed05\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") "
Jul 07 11:51:54.442751 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.442635    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e594d-e47a-4d1d-b85c-a3334539ed05-proxy-tls\") pod \"412e594d-e47a-4d1d-b85c-a3334539ed05\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") "
Jul 07 11:51:54.442751 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.442689    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdlfb\" (UniqueName: \"kubernetes.io/projected/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-api-access-bdlfb\") pod \"412e594d-e47a-4d1d-b85c-a3334539ed05\" (UID: \"412e594d-e47a-4d1d-b85c-a3334539ed05\") "
Jul 07 11:51:54.442930 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.442907    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "412e594d-e47a-4d1d-b85c-a3334539ed05" (UID: "412e594d-e47a-4d1d-b85c-a3334539ed05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:51:54.444779 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.444749    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-api-access-bdlfb" (OuterVolumeSpecName: "kube-api-access-bdlfb") pod "412e594d-e47a-4d1d-b85c-a3334539ed05" (UID: "412e594d-e47a-4d1d-b85c-a3334539ed05"). InnerVolumeSpecName "kube-api-access-bdlfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:51:54.444867 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.444777    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412e594d-e47a-4d1d-b85c-a3334539ed05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "412e594d-e47a-4d1d-b85c-a3334539ed05" (UID: "412e594d-e47a-4d1d-b85c-a3334539ed05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:51:54.543733 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.543702    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:51:54.543733 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.543730    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e594d-e47a-4d1d-b85c-a3334539ed05-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:51:54.543900 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:54.543743    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdlfb\" (UniqueName: \"kubernetes.io/projected/412e594d-e47a-4d1d-b85c-a3334539ed05-kube-api-access-bdlfb\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:51:55.159477 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.159443    2572 generic.go:356] "Generic (PLEG): container finished" podID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerID="e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58" exitCode=0
Jul 07 11:51:55.159832 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.159509    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"
Jul 07 11:51:55.159832 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.159534    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" event={"ID":"412e594d-e47a-4d1d-b85c-a3334539ed05","Type":"ContainerDied","Data":"e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58"}
Jul 07 11:51:55.159832 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.159621    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l" event={"ID":"412e594d-e47a-4d1d-b85c-a3334539ed05","Type":"ContainerDied","Data":"559592dc48fcb0f48f4e5bfcbccaae2ac64fe705cb938db82a7d7fe5a40a3d37"}
Jul 07 11:51:55.159832 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.159649    2572 scope.go:117] "RemoveContainer" containerID="41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31"
Jul 07 11:51:55.168192 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.168177    2572 scope.go:117] "RemoveContainer" containerID="e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58"
Jul 07 11:51:55.175265 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.175244    2572 scope.go:117] "RemoveContainer" containerID="41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31"
Jul 07 11:51:55.175512 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:51:55.175494    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31\": container with ID starting with 41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31 not found: ID does not exist" containerID="41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31"
Jul 07 11:51:55.175573 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.175519    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31"} err="failed to get container status \"41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31\": rpc error: code = NotFound desc = could not find container \"41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31\": container with ID starting with 41607902e8394f0dd4a0820491a6f5c83680ec78915540b30516c57c5b0b2b31 not found: ID does not exist"
Jul 07 11:51:55.175573 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.175536    2572 scope.go:117] "RemoveContainer" containerID="e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58"
Jul 07 11:51:55.175789 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:51:55.175772    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58\": container with ID starting with e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58 not found: ID does not exist" containerID="e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58"
Jul 07 11:51:55.175844 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.175793    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58"} err="failed to get container status \"e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58\": rpc error: code = NotFound desc = could not find container \"e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58\": container with ID starting with e4200f66d0ae722a0f6201edd1d94b9f5b5981bcdfe99020927d3f0e86dc5f58 not found: ID does not exist"
Jul 07 11:51:55.184074 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.184050    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"]
Jul 07 11:51:55.196391 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:55.196367    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l"]
Jul 07 11:51:56.116853 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:56.116817    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:56.438873 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:56.438786    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" path="/var/lib/kubelet/pods/412e594d-e47a-4d1d-b85c-a3334539ed05/volumes"
Jul 07 11:51:57.434194 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:57.434148    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:51:57.860225 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:57.860192    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:51:59.157766 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:59.157741    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:51:59.158201 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:51:59.158176    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 11:52:01.116622 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:01.116577    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:01.921518 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:01.921493    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:52:04.438229 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:04.438197    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:52:06.117023 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:06.116948    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:06.117353 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:06.117061    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:52:07.434643 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:07.434604    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:52:09.158167 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:09.158133    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 11:52:11.117364 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:11.117328    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:16.116809 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:16.116765    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:19.158893 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:19.158855    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 11:52:21.116870 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:21.116832    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:21.963452 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:21.963431    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:52:22.054931 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.054907    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls\") pod \"3cb373f2-d224-465e-8125-a85cef86ffb8\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") "
Jul 07 11:52:22.055039 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.054938    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb373f2-d224-465e-8125-a85cef86ffb8-openshift-service-ca-bundle\") pod \"3cb373f2-d224-465e-8125-a85cef86ffb8\" (UID: \"3cb373f2-d224-465e-8125-a85cef86ffb8\") "
Jul 07 11:52:22.055277 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.055253    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb373f2-d224-465e-8125-a85cef86ffb8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3cb373f2-d224-465e-8125-a85cef86ffb8" (UID: "3cb373f2-d224-465e-8125-a85cef86ffb8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:22.056829 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.056802    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cb373f2-d224-465e-8125-a85cef86ffb8" (UID: "3cb373f2-d224-465e-8125-a85cef86ffb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:52:22.156023 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.155977    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb373f2-d224-465e-8125-a85cef86ffb8-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:22.156023 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.155998    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb373f2-d224-465e-8125-a85cef86ffb8-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:22.276876 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.276852    2572 generic.go:356] "Generic (PLEG): container finished" podID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerID="66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd" exitCode=0
Jul 07 11:52:22.276958 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.276909    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"
Jul 07 11:52:22.277016 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.276908    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" event={"ID":"3cb373f2-d224-465e-8125-a85cef86ffb8","Type":"ContainerDied","Data":"66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd"}
Jul 07 11:52:22.277016 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.276998    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr" event={"ID":"3cb373f2-d224-465e-8125-a85cef86ffb8","Type":"ContainerDied","Data":"6a24a5a8cbf2aa0c5bf4d9ab0eb3084b678e7e3fc9f53671a1db970bb5a271e1"}
Jul 07 11:52:22.277090 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.277021    2572 scope.go:117] "RemoveContainer" containerID="66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd"
Jul 07 11:52:22.285230 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.285211    2572 scope.go:117] "RemoveContainer" containerID="66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd"
Jul 07 11:52:22.285470 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:22.285451    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd\": container with ID starting with 66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd not found: ID does not exist" containerID="66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd"
Jul 07 11:52:22.285540 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.285479    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd"} err="failed to get container status \"66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd\": rpc error: code = NotFound desc = could not find container \"66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd\": container with ID starting with 66c5fcfa26703b31616506ce7bfa9263abf62cf77b90a8205f604fd47c38ebcd not found: ID does not exist"
Jul 07 11:52:22.298066 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.298043    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"]
Jul 07 11:52:22.301631 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.301611    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr"]
Jul 07 11:52:22.438575 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:22.438526    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" path="/var/lib/kubelet/pods/3cb373f2-d224-465e-8125-a85cef86ffb8/volumes"
Jul 07 11:52:29.158968 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:29.158933    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.234808    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"]
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235558    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235578    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235606    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235616    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235686    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kube-rbac-proxy"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235696    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kube-rbac-proxy"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235834    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb373f2-d224-465e-8125-a85cef86ffb8" containerName="switch-graph-7cc25"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235853    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kube-rbac-proxy"
Jul 07 11:52:31.236386 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.235864    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="412e594d-e47a-4d1d-b85c-a3334539ed05" containerName="kserve-container"
Jul 07 11:52:31.240348 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.240317    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:31.242501 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.242479    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:31.243298 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.243277    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"]
Jul 07 11:52:31.245126 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.245106    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 07 11:52:31.245126 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.245118    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 07 11:52:31.423380 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.423357    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls\") pod \"model-chainer-6d7dd4967-4kmd4\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") " pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:31.423476 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.423398    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e202867-bd2c-4267-9d19-57da86c2f176-openshift-service-ca-bundle\") pod \"model-chainer-6d7dd4967-4kmd4\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") " pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:31.523695 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.523652    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls\") pod \"model-chainer-6d7dd4967-4kmd4\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") " pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:31.523798 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.523726    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e202867-bd2c-4267-9d19-57da86c2f176-openshift-service-ca-bundle\") pod \"model-chainer-6d7dd4967-4kmd4\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") " pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:31.523798 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:31.523784    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 07 11:52:31.523909 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:31.523846    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls podName:2e202867-bd2c-4267-9d19-57da86c2f176 nodeName:}" failed. No retries permitted until 2026-07-07 11:52:32.02383086 +0000 UTC m=+738.219407861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls") pod "model-chainer-6d7dd4967-4kmd4" (UID: "2e202867-bd2c-4267-9d19-57da86c2f176") : secret "model-chainer-serving-cert" not found
Jul 07 11:52:31.524309 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.524288    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e202867-bd2c-4267-9d19-57da86c2f176-openshift-service-ca-bundle\") pod \"model-chainer-6d7dd4967-4kmd4\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") " pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:31.754447 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:31.754427    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:32.026959 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:32.026938    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls\") pod \"model-chainer-6d7dd4967-4kmd4\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") " pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:32.029276 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:32.029259    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls\") pod \"model-chainer-6d7dd4967-4kmd4\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") " pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:32.154630 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:32.154590    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:32.279719 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:32.279696    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"]
Jul 07 11:52:32.282140 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:52:32.282115    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e202867_bd2c_4267_9d19_57da86c2f176.slice/crio-08f10426d70e93bea38fff173110925c8263583780dcbe3b89da431b2eee85d9 WatchSource:0}: Error finding container 08f10426d70e93bea38fff173110925c8263583780dcbe3b89da431b2eee85d9: Status 404 returned error can't find the container with id 08f10426d70e93bea38fff173110925c8263583780dcbe3b89da431b2eee85d9
Jul 07 11:52:32.317393 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:32.317371    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" event={"ID":"2e202867-bd2c-4267-9d19-57da86c2f176","Type":"ContainerStarted","Data":"08f10426d70e93bea38fff173110925c8263583780dcbe3b89da431b2eee85d9"}
Jul 07 11:52:33.322552 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:33.322520    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" event={"ID":"2e202867-bd2c-4267-9d19-57da86c2f176","Type":"ContainerStarted","Data":"c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79"}
Jul 07 11:52:33.322916 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:33.322636    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:33.340916 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:33.340872    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" podStartSLOduration=2.34085675 podStartE2EDuration="2.34085675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:52:33.339449341 +0000 UTC m=+739.535026363" watchObservedRunningTime="2026-07-07 11:52:33.34085675 +0000 UTC m=+739.536433771"
Jul 07 11:52:39.158805 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:39.158776    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 11:52:39.331560 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:39.331537    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:41.323908 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.323872    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"]
Jul 07 11:52:41.324354 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.324038    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer" containerID="cri-o://c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79" gracePeriod=30
Jul 07 11:52:41.347761 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.347737    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:41.432883 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.432852    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"]
Jul 07 11:52:41.433118 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.433097    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" containerID="cri-o://ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454" gracePeriod=30
Jul 07 11:52:41.433193 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.433091    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kube-rbac-proxy" containerID="cri-o://6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4" gracePeriod=30
Jul 07 11:52:41.460109 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.460082    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:41.495035 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.495013    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"]
Jul 07 11:52:41.495252 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.495232    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container" containerID="cri-o://c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b" gracePeriod=30
Jul 07 11:52:41.495329 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.495309    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kube-rbac-proxy" containerID="cri-o://65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b" gracePeriod=30
Jul 07 11:52:41.505967 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.505946    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"]
Jul 07 11:52:41.509500 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.509485    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.511747 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.511726    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-07b43-predictor-serving-cert\""
Jul 07 11:52:41.511845 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.511766    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-07b43-kube-rbac-proxy-sar-config\""
Jul 07 11:52:41.519445 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.519421    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"]
Jul 07 11:52:41.541927 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.541907    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:41.581781 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.581722    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"]
Jul 07 11:52:41.581964 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.581940    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" containerID="cri-o://78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009" gracePeriod=30
Jul 07 11:52:41.582029 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.581994    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kube-rbac-proxy" containerID="cri-o://194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448" gracePeriod=30
Jul 07 11:52:41.628083 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.628054    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:41.696816 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.696797    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7psn\" (UniqueName: \"kubernetes.io/projected/1156089a-302d-410d-a777-37f2860a1f95-kube-api-access-g7psn\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.696893 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.696852    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1156089a-302d-410d-a777-37f2860a1f95-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.696936 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.696906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1156089a-302d-410d-a777-37f2860a1f95-proxy-tls\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.725384 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.725360    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:41.797636 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.797613    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1156089a-302d-410d-a777-37f2860a1f95-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.797781 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.797649    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1156089a-302d-410d-a777-37f2860a1f95-proxy-tls\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.797781 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.797712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7psn\" (UniqueName: \"kubernetes.io/projected/1156089a-302d-410d-a777-37f2860a1f95-kube-api-access-g7psn\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.798330 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.798309    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1156089a-302d-410d-a777-37f2860a1f95-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.799958 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.799934    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1156089a-302d-410d-a777-37f2860a1f95-proxy-tls\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.806244 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.806221    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7psn\" (UniqueName: \"kubernetes.io/projected/1156089a-302d-410d-a777-37f2860a1f95-kube-api-access-g7psn\") pod \"success-200-isvc-07b43-predictor-7b997f5f8f-q59st\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") " pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.820859 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.820838    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:41.916579 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.916542    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 07 11:52:41.921221 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.921189    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 11:52:41.955012 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:41.954987    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"]
Jul 07 11:52:41.957319 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:52:41.957287    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1156089a_302d_410d_a777_37f2860a1f95.slice/crio-55262f903d3751eb2201b17d1f1e4101b190764bb4315f7bb41856573cf8b7a0 WatchSource:0}: Error finding container 55262f903d3751eb2201b17d1f1e4101b190764bb4315f7bb41856573cf8b7a0: Status 404 returned error can't find the container with id 55262f903d3751eb2201b17d1f1e4101b190764bb4315f7bb41856573cf8b7a0
Jul 07 11:52:42.140584 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.140508    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:42.361920 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.361887    2572 generic.go:356] "Generic (PLEG): container finished" podID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerID="6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4" exitCode=2
Jul 07 11:52:42.362296 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.361962    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" event={"ID":"e19a862e-90b6-4fb6-aa63-6f430ce4652e","Type":"ContainerDied","Data":"6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4"}
Jul 07 11:52:42.363899 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.363879    2572 generic.go:356] "Generic (PLEG): container finished" podID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerID="194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448" exitCode=2
Jul 07 11:52:42.364010 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.363951    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" event={"ID":"e7a85d9c-f4b1-44f7-a758-bfdaca67607b","Type":"ContainerDied","Data":"194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448"}
Jul 07 11:52:42.365829 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.365812    2572 generic.go:356] "Generic (PLEG): container finished" podID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerID="65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b" exitCode=2
Jul 07 11:52:42.365927 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.365875    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" event={"ID":"d638385e-c8ef-4f64-a1f9-d34e94619741","Type":"ContainerDied","Data":"65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b"}
Jul 07 11:52:42.367366 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.367336    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" event={"ID":"1156089a-302d-410d-a777-37f2860a1f95","Type":"ContainerStarted","Data":"dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67"}
Jul 07 11:52:42.367452 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.367378    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" event={"ID":"1156089a-302d-410d-a777-37f2860a1f95","Type":"ContainerStarted","Data":"fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed"}
Jul 07 11:52:42.367452 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.367406    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" event={"ID":"1156089a-302d-410d-a777-37f2860a1f95","Type":"ContainerStarted","Data":"55262f903d3751eb2201b17d1f1e4101b190764bb4315f7bb41856573cf8b7a0"}
Jul 07 11:52:42.367452 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.367429    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:42.367582 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.367478    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:42.368640 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.368619    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:52:42.384348 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.384310    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podStartSLOduration=1.384298167 podStartE2EDuration="1.384298167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:52:42.383171588 +0000 UTC m=+748.578748623" watchObservedRunningTime="2026-07-07 11:52:42.384298167 +0000 UTC m=+748.579875188"
Jul 07 11:52:42.854718 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:42.854673    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 07 11:52:43.371751 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:43.371712    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:52:43.858758 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:43.858716    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 07 11:52:44.332502 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:44.332462    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:44.435753 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:44.435701    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:52:44.751110 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:44.751082    2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a85d9c_f4b1_44f7_a758_bfdaca67607b.slice/crio-78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009.scope\": RecentStats: unable to find data in memory cache]"
Jul 07 11:52:44.932020 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:44.931998    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:52:45.126838 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.126812    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kserve-provision-location\") pod \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") "
Jul 07 11:52:45.126977 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.126866    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh444\" (UniqueName: \"kubernetes.io/projected/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-api-access-zh444\") pod \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") "
Jul 07 11:52:45.126977 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.126887    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-cabundle-cert\") pod \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") "
Jul 07 11:52:45.126977 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.126907    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-rbac-proxy-sar-config\") pod \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") "
Jul 07 11:52:45.126977 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.126938    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-proxy-tls\") pod \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\" (UID: \"e7a85d9c-f4b1-44f7-a758-bfdaca67607b\") "
Jul 07 11:52:45.127232 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.127136    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7a85d9c-f4b1-44f7-a758-bfdaca67607b" (UID: "e7a85d9c-f4b1-44f7-a758-bfdaca67607b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:52:45.127275 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.127235    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7a85d9c-f4b1-44f7-a758-bfdaca67607b" (UID: "e7a85d9c-f4b1-44f7-a758-bfdaca67607b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:45.127316 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.127273    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7a85d9c-f4b1-44f7-a758-bfdaca67607b" (UID: "e7a85d9c-f4b1-44f7-a758-bfdaca67607b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:45.129053 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.129024    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7a85d9c-f4b1-44f7-a758-bfdaca67607b" (UID: "e7a85d9c-f4b1-44f7-a758-bfdaca67607b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:52:45.129210 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.129192    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-api-access-zh444" (OuterVolumeSpecName: "kube-api-access-zh444") pod "e7a85d9c-f4b1-44f7-a758-bfdaca67607b" (UID: "e7a85d9c-f4b1-44f7-a758-bfdaca67607b"). InnerVolumeSpecName "kube-api-access-zh444". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:52:45.228282 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.228255    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh444\" (UniqueName: \"kubernetes.io/projected/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-api-access-zh444\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:45.228282 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.228278    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-cabundle-cert\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:45.228399 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.228288    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:45.228399 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.228298    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:45.228399 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.228306    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7a85d9c-f4b1-44f7-a758-bfdaca67607b-kserve-provision-location\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:45.381953 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.381864    2572 generic.go:356] "Generic (PLEG): container finished" podID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerID="78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009" exitCode=0
Jul 07 11:52:45.381953 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.381912    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" event={"ID":"e7a85d9c-f4b1-44f7-a758-bfdaca67607b","Type":"ContainerDied","Data":"78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009"}
Jul 07 11:52:45.381953 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.381922    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"
Jul 07 11:52:45.382200 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.381961    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2" event={"ID":"e7a85d9c-f4b1-44f7-a758-bfdaca67607b","Type":"ContainerDied","Data":"71ab62e07ead73fefa98f145c38022deacac9a2346260138f9287bb8dd6d3a7d"}
Jul 07 11:52:45.382200 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.381986    2572 scope.go:117] "RemoveContainer" containerID="194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448"
Jul 07 11:52:45.390356 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.390342    2572 scope.go:117] "RemoveContainer" containerID="78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009"
Jul 07 11:52:45.397432 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.397418    2572 scope.go:117] "RemoveContainer" containerID="0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6"
Jul 07 11:52:45.405297 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.405280    2572 scope.go:117] "RemoveContainer" containerID="194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448"
Jul 07 11:52:45.405574 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:45.405551    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448\": container with ID starting with 194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448 not found: ID does not exist" containerID="194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448"
Jul 07 11:52:45.405627 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.405582    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448"} err="failed to get container status \"194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448\": rpc error: code = NotFound desc = could not find container \"194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448\": container with ID starting with 194de512a69fb91baee74f35f864dbdf76d9478dee57d5eb1ebe691eea554448 not found: ID does not exist"
Jul 07 11:52:45.405627 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.405599    2572 scope.go:117] "RemoveContainer" containerID="78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009"
Jul 07 11:52:45.405846 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.405828    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"]
Jul 07 11:52:45.405907 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:45.405850    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009\": container with ID starting with 78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009 not found: ID does not exist" containerID="78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009"
Jul 07 11:52:45.405907 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.405878    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009"} err="failed to get container status \"78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009\": rpc error: code = NotFound desc = could not find container \"78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009\": container with ID starting with 78dad34e7c99bb5b655f13afe62eb3da2c6f3e60381444c4b9314444b9013009 not found: ID does not exist"
Jul 07 11:52:45.405907 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.405900    2572 scope.go:117] "RemoveContainer" containerID="0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6"
Jul 07 11:52:45.406149 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:45.406133    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6\": container with ID starting with 0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6 not found: ID does not exist" containerID="0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6"
Jul 07 11:52:45.406185 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.406153    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6"} err="failed to get container status \"0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6\": rpc error: code = NotFound desc = could not find container \"0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6\": container with ID starting with 0bbee42f0c3615160eceec480461d4d6813f673840bb0397f1996cc650204af6 not found: ID does not exist"
Jul 07 11:52:45.409755 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:45.409737    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2"]
Jul 07 11:52:46.180051 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.179998    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:52:46.182586 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.182567    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:52:46.235157 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235108    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d638385e-c8ef-4f64-a1f9-d34e94619741-kserve-provision-location\") pod \"d638385e-c8ef-4f64-a1f9-d34e94619741\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") "
Jul 07 11:52:46.235270 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235185    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls\") pod \"d638385e-c8ef-4f64-a1f9-d34e94619741\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") "
Jul 07 11:52:46.235270 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235212    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls\") pod \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") "
Jul 07 11:52:46.235270 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235239    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-rbac-proxy-sar-config\") pod \"d638385e-c8ef-4f64-a1f9-d34e94619741\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") "
Jul 07 11:52:46.235270 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235268    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rns7\" (UniqueName: \"kubernetes.io/projected/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-api-access-2rns7\") pod \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") "
Jul 07 11:52:46.235840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235299    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wspxr\" (UniqueName: \"kubernetes.io/projected/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-api-access-wspxr\") pod \"d638385e-c8ef-4f64-a1f9-d34e94619741\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") "
Jul 07 11:52:46.235840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235324    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-cabundle-cert\") pod \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") "
Jul 07 11:52:46.235840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235373    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-cabundle-cert\") pod \"d638385e-c8ef-4f64-a1f9-d34e94619741\" (UID: \"d638385e-c8ef-4f64-a1f9-d34e94619741\") "
Jul 07 11:52:46.235840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235397    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kserve-provision-location\") pod \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") "
Jul 07 11:52:46.235840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235407    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d638385e-c8ef-4f64-a1f9-d34e94619741-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d638385e-c8ef-4f64-a1f9-d34e94619741" (UID: "d638385e-c8ef-4f64-a1f9-d34e94619741"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:52:46.235840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235466    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-rbac-proxy-sar-config\") pod \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\" (UID: \"e19a862e-90b6-4fb6-aa63-6f430ce4652e\") "
Jul 07 11:52:46.235840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235712    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d638385e-c8ef-4f64-a1f9-d34e94619741-kserve-provision-location\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.235840 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235763    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d638385e-c8ef-4f64-a1f9-d34e94619741" (UID: "d638385e-c8ef-4f64-a1f9-d34e94619741"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:46.236364 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.235979    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d638385e-c8ef-4f64-a1f9-d34e94619741" (UID: "d638385e-c8ef-4f64-a1f9-d34e94619741"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:46.236364 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.236110    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e19a862e-90b6-4fb6-aa63-6f430ce4652e" (UID: "e19a862e-90b6-4fb6-aa63-6f430ce4652e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:46.236364 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.236140    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e19a862e-90b6-4fb6-aa63-6f430ce4652e" (UID: "e19a862e-90b6-4fb6-aa63-6f430ce4652e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:52:46.236364 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.236173    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e19a862e-90b6-4fb6-aa63-6f430ce4652e" (UID: "e19a862e-90b6-4fb6-aa63-6f430ce4652e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:46.238499 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.238472    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-api-access-wspxr" (OuterVolumeSpecName: "kube-api-access-wspxr") pod "d638385e-c8ef-4f64-a1f9-d34e94619741" (UID: "d638385e-c8ef-4f64-a1f9-d34e94619741"). InnerVolumeSpecName "kube-api-access-wspxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:52:46.238611 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.238506    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-api-access-2rns7" (OuterVolumeSpecName: "kube-api-access-2rns7") pod "e19a862e-90b6-4fb6-aa63-6f430ce4652e" (UID: "e19a862e-90b6-4fb6-aa63-6f430ce4652e"). InnerVolumeSpecName "kube-api-access-2rns7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:52:46.238611 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.238520    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e19a862e-90b6-4fb6-aa63-6f430ce4652e" (UID: "e19a862e-90b6-4fb6-aa63-6f430ce4652e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:52:46.238611 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.238524    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d638385e-c8ef-4f64-a1f9-d34e94619741" (UID: "d638385e-c8ef-4f64-a1f9-d34e94619741"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:52:46.336508 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336484    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d638385e-c8ef-4f64-a1f9-d34e94619741-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.336508 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336503    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19a862e-90b6-4fb6-aa63-6f430ce4652e-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.336687 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336513    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.336687 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336523    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rns7\" (UniqueName: \"kubernetes.io/projected/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-api-access-2rns7\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.336687 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336532    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wspxr\" (UniqueName: \"kubernetes.io/projected/d638385e-c8ef-4f64-a1f9-d34e94619741-kube-api-access-wspxr\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.336687 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336539    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-cabundle-cert\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.336687 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336547    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d638385e-c8ef-4f64-a1f9-d34e94619741-cabundle-cert\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.336687 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336555    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kserve-provision-location\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.336687 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.336564    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19a862e-90b6-4fb6-aa63-6f430ce4652e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:52:46.387498 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.387476    2572 generic.go:356] "Generic (PLEG): container finished" podID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerID="c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b" exitCode=0
Jul 07 11:52:46.387573 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.387536    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"
Jul 07 11:52:46.387611 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.387563    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" event={"ID":"d638385e-c8ef-4f64-a1f9-d34e94619741","Type":"ContainerDied","Data":"c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b"}
Jul 07 11:52:46.387699 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.387681    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x" event={"ID":"d638385e-c8ef-4f64-a1f9-d34e94619741","Type":"ContainerDied","Data":"f07f905f0986ca648b5abe4267f365f673e8226c625189a074b6bf65ae26d221"}
Jul 07 11:52:46.387745 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.387730    2572 scope.go:117] "RemoveContainer" containerID="65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b"
Jul 07 11:52:46.389510 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.389492    2572 generic.go:356] "Generic (PLEG): container finished" podID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerID="ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454" exitCode=0
Jul 07 11:52:46.389606 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.389550    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" event={"ID":"e19a862e-90b6-4fb6-aa63-6f430ce4652e","Type":"ContainerDied","Data":"ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454"}
Jul 07 11:52:46.389606 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.389571    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"
Jul 07 11:52:46.389738 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.389606    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp" event={"ID":"e19a862e-90b6-4fb6-aa63-6f430ce4652e","Type":"ContainerDied","Data":"d2f42d8ab2ad18bad8d823b3f9c06bdd57e4f42697ab178b0cc7d0440619fa3a"}
Jul 07 11:52:46.396902 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.396885    2572 scope.go:117] "RemoveContainer" containerID="c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b"
Jul 07 11:52:46.404484 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.404465    2572 scope.go:117] "RemoveContainer" containerID="4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274"
Jul 07 11:52:46.411071 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.411056    2572 scope.go:117] "RemoveContainer" containerID="65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b"
Jul 07 11:52:46.411290 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:46.411275    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b\": container with ID starting with 65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b not found: ID does not exist" containerID="65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b"
Jul 07 11:52:46.411328 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.411296    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b"} err="failed to get container status \"65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b\": rpc error: code = NotFound desc = could not find container \"65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b\": container with ID starting with 65a70cb79b857893a18d9dd2b8d25f1bfa8fc2c32280833db037bee2b5d8716b not found: ID does not exist"
Jul 07 11:52:46.411328 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.411310    2572 scope.go:117] "RemoveContainer" containerID="c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b"
Jul 07 11:52:46.411518 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:46.411502    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b\": container with ID starting with c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b not found: ID does not exist" containerID="c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b"
Jul 07 11:52:46.411557 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.411524    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b"} err="failed to get container status \"c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b\": rpc error: code = NotFound desc = could not find container \"c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b\": container with ID starting with c67c8f31718f472856ddfd3d4dbf9a09a5b37d7374fef104666d425657487b6b not found: ID does not exist"
Jul 07 11:52:46.411557 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.411540    2572 scope.go:117] "RemoveContainer" containerID="4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274"
Jul 07 11:52:46.411762 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:46.411747    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274\": container with ID starting with 4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274 not found: ID does not exist" containerID="4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274"
Jul 07 11:52:46.411811 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.411765    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274"} err="failed to get container status \"4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274\": rpc error: code = NotFound desc = could not find container \"4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274\": container with ID starting with 4818598fb0ed928ad41fa0bbd2dce75febacab72f41c8d4eff7665a1b7787274 not found: ID does not exist"
Jul 07 11:52:46.411811 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.411778    2572 scope.go:117] "RemoveContainer" containerID="6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4"
Jul 07 11:52:46.414615 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.414594    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"]
Jul 07 11:52:46.418650 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.418637    2572 scope.go:117] "RemoveContainer" containerID="ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454"
Jul 07 11:52:46.421852 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.421834    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x"]
Jul 07 11:52:46.425673 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.425646    2572 scope.go:117] "RemoveContainer" containerID="ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706"
Jul 07 11:52:46.433578 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.433544    2572 scope.go:117] "RemoveContainer" containerID="6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4"
Jul 07 11:52:46.434455 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:46.434432    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4\": container with ID starting with 6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4 not found: ID does not exist" containerID="6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4"
Jul 07 11:52:46.434557 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.434463    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4"} err="failed to get container status \"6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4\": rpc error: code = NotFound desc = could not find container \"6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4\": container with ID starting with 6f6c97c4e8f593965dee37bdc33dbcdc4831a1d7830769fdd7b299e124405fd4 not found: ID does not exist"
Jul 07 11:52:46.434557 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.434506    2572 scope.go:117] "RemoveContainer" containerID="ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454"
Jul 07 11:52:46.434854 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:46.434831    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454\": container with ID starting with ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454 not found: ID does not exist" containerID="ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454"
Jul 07 11:52:46.434931 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.434863    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454"} err="failed to get container status \"ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454\": rpc error: code = NotFound desc = could not find container \"ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454\": container with ID starting with ec2446dd301ec32f159e89f0eaa3514633fc3dc5464f6b809d958ae4d0ca2454 not found: ID does not exist"
Jul 07 11:52:46.434931 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.434884    2572 scope.go:117] "RemoveContainer" containerID="ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706"
Jul 07 11:52:46.435146 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:46.435114    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706\": container with ID starting with ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706 not found: ID does not exist" containerID="ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706"
Jul 07 11:52:46.435244 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.435143    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706"} err="failed to get container status \"ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706\": rpc error: code = NotFound desc = could not find container \"ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706\": container with ID starting with ca745acc5131d459c82665045f3e5e58e49ac8caecec4f28c33c61e17348e706 not found: ID does not exist"
Jul 07 11:52:46.440250 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.440228    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" path="/var/lib/kubelet/pods/d638385e-c8ef-4f64-a1f9-d34e94619741/volumes"
Jul 07 11:52:46.440763 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.440750    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" path="/var/lib/kubelet/pods/e7a85d9c-f4b1-44f7-a758-bfdaca67607b/volumes"
Jul 07 11:52:46.441074 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.441062    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"]
Jul 07 11:52:46.441107 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:46.441088    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp"]
Jul 07 11:52:48.376923 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:48.376893    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:52:48.377338 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:48.377314    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:52:48.438279 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:48.438252    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" path="/var/lib/kubelet/pods/e19a862e-90b6-4fb6-aa63-6f430ce4652e/volumes"
Jul 07 11:52:49.331723 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:49.331651    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:51.509039 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509008    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"]
Jul 07 11:52:51.509383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509321    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="storage-initializer"
Jul 07 11:52:51.509383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509332    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="storage-initializer"
Jul 07 11:52:51.509383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509341    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container"
Jul 07 11:52:51.509383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509347    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container"
Jul 07 11:52:51.509383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509364    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="storage-initializer"
Jul 07 11:52:51.509383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509369    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="storage-initializer"
Jul 07 11:52:51.509383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509377    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509383 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509382    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509390    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509395    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509402    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509407    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509420    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="storage-initializer"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509425    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="storage-initializer"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509432    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509437    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509444    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509449    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509492    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509499    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a85d9c-f4b1-44f7-a758-bfdaca67607b" containerName="kserve-container"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509506    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kserve-container"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509512    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d638385e-c8ef-4f64-a1f9-d34e94619741" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509517    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kube-rbac-proxy"
Jul 07 11:52:51.509614 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.509522    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19a862e-90b6-4fb6-aa63-6f430ce4652e" containerName="kserve-container"
Jul 07 11:52:51.513895 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.513876    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:51.516145 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.516125    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-5e1d2-serving-cert\""
Jul 07 11:52:51.516241 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.516125    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-5e1d2-kube-rbac-proxy-sar-config\""
Jul 07 11:52:51.522832 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.522811    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"]
Jul 07 11:52:51.528545 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.528522    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:51.572146 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.572117    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8235a072-409a-495b-bd9e-0a35ef4c27e0-openshift-service-ca-bundle\") pod \"switch-graph-5e1d2-69cf5bd554-gfrts\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") " pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:51.572223 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.572157    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls\") pod \"switch-graph-5e1d2-69cf5bd554-gfrts\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") " pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:51.673013 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.672987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls\") pod \"switch-graph-5e1d2-69cf5bd554-gfrts\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") " pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:51.673087 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.673046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8235a072-409a-495b-bd9e-0a35ef4c27e0-openshift-service-ca-bundle\") pod \"switch-graph-5e1d2-69cf5bd554-gfrts\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") " pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:51.673131 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:51.673117    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-5e1d2-serving-cert: secret "switch-graph-5e1d2-serving-cert" not found
Jul 07 11:52:51.673202 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:52:51.673192    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls podName:8235a072-409a-495b-bd9e-0a35ef4c27e0 nodeName:}" failed. No retries permitted until 2026-07-07 11:52:52.173177318 +0000 UTC m=+758.368754318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls") pod "switch-graph-5e1d2-69cf5bd554-gfrts" (UID: "8235a072-409a-495b-bd9e-0a35ef4c27e0") : secret "switch-graph-5e1d2-serving-cert" not found
Jul 07 11:52:51.673565 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.673549    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8235a072-409a-495b-bd9e-0a35ef4c27e0-openshift-service-ca-bundle\") pod \"switch-graph-5e1d2-69cf5bd554-gfrts\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") " pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:51.734531 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:51.734499    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:52:52.176194 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:52.176161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls\") pod \"switch-graph-5e1d2-69cf5bd554-gfrts\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") " pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:52.178595 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:52.178561    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls\") pod \"switch-graph-5e1d2-69cf5bd554-gfrts\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") " pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:52.423746 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:52.423720    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:52.547060 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:52.547037    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"]
Jul 07 11:52:52.549223 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:52:52.549197    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8235a072_409a_495b_bd9e_0a35ef4c27e0.slice/crio-4b750a8d2ade4fc5ecd027fa510b98ab6d3577f2fe572adb2480e01f2b9e3f40 WatchSource:0}: Error finding container 4b750a8d2ade4fc5ecd027fa510b98ab6d3577f2fe572adb2480e01f2b9e3f40: Status 404 returned error can't find the container with id 4b750a8d2ade4fc5ecd027fa510b98ab6d3577f2fe572adb2480e01f2b9e3f40
Jul 07 11:52:53.423418 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:53.423381    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" event={"ID":"8235a072-409a-495b-bd9e-0a35ef4c27e0","Type":"ContainerStarted","Data":"326f3b2414b305b213e354dbe2a59a6c702d339e58dd4a254578ce623c1f6066"}
Jul 07 11:52:53.423559 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:53.423431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" event={"ID":"8235a072-409a-495b-bd9e-0a35ef4c27e0","Type":"ContainerStarted","Data":"4b750a8d2ade4fc5ecd027fa510b98ab6d3577f2fe572adb2480e01f2b9e3f40"}
Jul 07 11:52:53.423559 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:53.423512    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:52:53.438886 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:53.438729    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" podStartSLOduration=2.438714916 podStartE2EDuration="2.438714916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:52:53.438433995 +0000 UTC m=+759.634011017" watchObservedRunningTime="2026-07-07 11:52:53.438714916 +0000 UTC m=+759.634291939"
Jul 07 11:52:54.331226 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:54.331195    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:54.331590 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:54.331284    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:52:58.378192 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:58.378156    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:52:59.332018 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:59.331961    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:59.433624 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:52:59.433600    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 11:53:04.331167 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:04.331133    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:07.973353 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:07.973317    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:53:08.185517 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:08.185484    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:53:08.377707 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:08.377677    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:53:09.330792 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:09.330756    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:11.462926 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.462897    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:53:11.493209 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.493184    2572 generic.go:356] "Generic (PLEG): container finished" podID="2e202867-bd2c-4267-9d19-57da86c2f176" containerID="c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79" exitCode=0
Jul 07 11:53:11.493302 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.493218    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"
Jul 07 11:53:11.493302 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.493272    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" event={"ID":"2e202867-bd2c-4267-9d19-57da86c2f176","Type":"ContainerDied","Data":"c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79"}
Jul 07 11:53:11.493385 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.493332    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4" event={"ID":"2e202867-bd2c-4267-9d19-57da86c2f176","Type":"ContainerDied","Data":"08f10426d70e93bea38fff173110925c8263583780dcbe3b89da431b2eee85d9"}
Jul 07 11:53:11.493385 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.493364    2572 scope.go:117] "RemoveContainer" containerID="c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79"
Jul 07 11:53:11.501178 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.501157    2572 scope.go:117] "RemoveContainer" containerID="c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79"
Jul 07 11:53:11.501439 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:53:11.501420    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79\": container with ID starting with c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79 not found: ID does not exist" containerID="c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79"
Jul 07 11:53:11.501482 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.501448    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79"} err="failed to get container status \"c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79\": rpc error: code = NotFound desc = could not find container \"c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79\": container with ID starting with c9ff2b1b2d9d8cecbd8cbcc550c1be44c26b4c296960295a4b2cab4ae469ca79 not found: ID does not exist"
Jul 07 11:53:11.520194 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.520177    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls\") pod \"2e202867-bd2c-4267-9d19-57da86c2f176\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") "
Jul 07 11:53:11.520279 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.520206    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e202867-bd2c-4267-9d19-57da86c2f176-openshift-service-ca-bundle\") pod \"2e202867-bd2c-4267-9d19-57da86c2f176\" (UID: \"2e202867-bd2c-4267-9d19-57da86c2f176\") "
Jul 07 11:53:11.521035 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.520653    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e202867-bd2c-4267-9d19-57da86c2f176-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2e202867-bd2c-4267-9d19-57da86c2f176" (UID: "2e202867-bd2c-4267-9d19-57da86c2f176"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:53:11.521138 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.521111    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e202867-bd2c-4267-9d19-57da86c2f176-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:53:11.522466 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.522446    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e202867-bd2c-4267-9d19-57da86c2f176" (UID: "2e202867-bd2c-4267-9d19-57da86c2f176"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:53:11.621418 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.621372    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e202867-bd2c-4267-9d19-57da86c2f176-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 11:53:11.813571 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.813548    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"]
Jul 07 11:53:11.817165 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:11.817146    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4"]
Jul 07 11:53:12.438829 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:12.438794    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" path="/var/lib/kubelet/pods/2e202867-bd2c-4267-9d19-57da86c2f176/volumes"
Jul 07 11:53:18.377598 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:18.377559    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:53:28.378845 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:28.378812    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 11:53:41.498884 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.498811    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"]
Jul 07 11:53:41.500682 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.499396    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer"
Jul 07 11:53:41.500682 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.499422    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer"
Jul 07 11:53:41.500682 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.499812    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e202867-bd2c-4267-9d19-57da86c2f176" containerName="model-chainer"
Jul 07 11:53:41.505637 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.505616    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:41.508224 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.508206    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-07b43-kube-rbac-proxy-sar-config\""
Jul 07 11:53:41.508307 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.508209    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-07b43-serving-cert\""
Jul 07 11:53:41.512495 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.512148    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"]
Jul 07 11:53:41.633414 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.633383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cdc2570-8db9-42ef-a873-cafecd654cc0-openshift-service-ca-bundle\") pod \"sequence-graph-07b43-dccbfccf8-fw476\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") " pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:41.633551 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.633446    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls\") pod \"sequence-graph-07b43-dccbfccf8-fw476\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") " pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:41.649608 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.649586    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:53:41.734256 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.734233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cdc2570-8db9-42ef-a873-cafecd654cc0-openshift-service-ca-bundle\") pod \"sequence-graph-07b43-dccbfccf8-fw476\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") " pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:41.734346 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.734279    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls\") pod \"sequence-graph-07b43-dccbfccf8-fw476\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") " pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:41.734388 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:53:41.734361    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-07b43-serving-cert: secret "sequence-graph-07b43-serving-cert" not found
Jul 07 11:53:41.734428 ip-10-0-140-177 kubenswrapper[2572]: E0707 11:53:41.734414    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls podName:1cdc2570-8db9-42ef-a873-cafecd654cc0 nodeName:}" failed. No retries permitted until 2026-07-07 11:53:42.234399236 +0000 UTC m=+808.429976236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls") pod "sequence-graph-07b43-dccbfccf8-fw476" (UID: "1cdc2570-8db9-42ef-a873-cafecd654cc0") : secret "sequence-graph-07b43-serving-cert" not found
Jul 07 11:53:41.734862 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.734845    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cdc2570-8db9-42ef-a873-cafecd654cc0-openshift-service-ca-bundle\") pod \"sequence-graph-07b43-dccbfccf8-fw476\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") " pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:41.980210 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:41.980138    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:53:42.238028 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:42.237957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls\") pod \"sequence-graph-07b43-dccbfccf8-fw476\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") " pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:42.240234 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:42.240205    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls\") pod \"sequence-graph-07b43-dccbfccf8-fw476\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") " pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:42.416348 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:42.416322    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:42.542676 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:42.542634    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"]
Jul 07 11:53:42.545267 ip-10-0-140-177 kubenswrapper[2572]: W0707 11:53:42.545235    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdc2570_8db9_42ef_a873_cafecd654cc0.slice/crio-cd0da7aa5f7b56700ce18aef632f25fdb57b27ae50b794569aa55c54921b2cfc WatchSource:0}: Error finding container cd0da7aa5f7b56700ce18aef632f25fdb57b27ae50b794569aa55c54921b2cfc: Status 404 returned error can't find the container with id cd0da7aa5f7b56700ce18aef632f25fdb57b27ae50b794569aa55c54921b2cfc
Jul 07 11:53:42.609388 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:42.609362    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" event={"ID":"1cdc2570-8db9-42ef-a873-cafecd654cc0","Type":"ContainerStarted","Data":"856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5"}
Jul 07 11:53:42.609515 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:42.609407    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" event={"ID":"1cdc2570-8db9-42ef-a873-cafecd654cc0","Type":"ContainerStarted","Data":"cd0da7aa5f7b56700ce18aef632f25fdb57b27ae50b794569aa55c54921b2cfc"}
Jul 07 11:53:42.609515 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:42.609498    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:53:42.628055 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:42.628015    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" podStartSLOduration=1.628001716 podStartE2EDuration="1.628001716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:53:42.627194316 +0000 UTC m=+808.822771331" watchObservedRunningTime="2026-07-07 11:53:42.628001716 +0000 UTC m=+808.823578737"
Jul 07 11:53:48.618838 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:53:48.618811    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 11:54:12.222014 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:54:12.221984    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:54:12.308118 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:54:12.308088    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:54:26.769013 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:54:26.768980    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:55:14.261894 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:55:14.261857    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:55:14.263126 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:55:14.263105    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 11:56:25.582223 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:56:25.582185    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:57:08.926429 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:57:08.926391    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:57:10.281260 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:57:10.281229    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 11:58:15.630701 ip-10-0-140-177 kubenswrapper[2572]: I0707 11:58:15.630650    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:00:14.287007 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:00:14.286975    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:00:14.288485 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:00:14.288466    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:01:06.206177 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.206104    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"]
Jul 07 12:01:06.206614 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.206317    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2" containerID="cri-o://326f3b2414b305b213e354dbe2a59a6c702d339e58dd4a254578ce623c1f6066" gracePeriod=30
Jul 07 12:01:06.394882 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.394854    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:06.540156 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.540129    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:06.628132 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.628108    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"]
Jul 07 12:01:06.631424 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.631403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:06.634819 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.634797    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-462dc-predictor-serving-cert\""
Jul 07 12:01:06.635928 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.635909    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-462dc-kube-rbac-proxy-sar-config\""
Jul 07 12:01:06.659123 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.659101    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"]
Jul 07 12:01:06.684583 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.684555    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfd9\" (UniqueName: \"kubernetes.io/projected/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-api-access-csfd9\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:06.684701 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.684593    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:06.684701 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.684644    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:06.698686 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.698650    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"]
Jul 07 12:01:06.698884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.698855    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container" containerID="cri-o://8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf" gracePeriod=30
Jul 07 12:01:06.699008 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.698888    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kube-rbac-proxy" containerID="cri-o://c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398" gracePeriod=30
Jul 07 12:01:06.741218 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.741194    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:06.785773 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.785750    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:06.785856 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.785786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:06.785856 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.785842    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfd9\" (UniqueName: \"kubernetes.io/projected/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-api-access-csfd9\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:06.785935 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:01:06.785884    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-462dc-predictor-serving-cert: secret "success-200-isvc-462dc-predictor-serving-cert" not found
Jul 07 12:01:06.785984 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:01:06.785936    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls podName:f705fc06-56a8-428f-926b-fd2a0b6530a1 nodeName:}" failed. No retries permitted until 2026-07-07 12:01:07.285923064 +0000 UTC m=+1253.481500064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls") pod "success-200-isvc-462dc-predictor-8d4845d7-rtcpm" (UID: "f705fc06-56a8-428f-926b-fd2a0b6530a1") : secret "success-200-isvc-462dc-predictor-serving-cert" not found
Jul 07 12:01:06.786460 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.786441    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:06.801915 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:06.801863    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfd9\" (UniqueName: \"kubernetes.io/projected/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-api-access-csfd9\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:07.221546 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:07.221469    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:07.287930 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:07.287894    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:07.290652 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:07.290624    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls\") pod \"success-200-isvc-462dc-predictor-8d4845d7-rtcpm\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") " pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:07.318125 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:07.318098    2572 generic.go:356] "Generic (PLEG): container finished" podID="34a35811-de8e-4483-b222-f574a9566720" containerID="c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398" exitCode=2
Jul 07 12:01:07.318263 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:07.318164    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" event={"ID":"34a35811-de8e-4483-b222-f574a9566720","Type":"ContainerDied","Data":"c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398"}
Jul 07 12:01:07.357327 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:07.357300    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:07.542253 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:07.542225    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:07.681189 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:07.681151    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"]
Jul 07 12:01:07.684279 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:01:07.684254    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf705fc06_56a8_428f_926b_fd2a0b6530a1.slice/crio-96494011e81a653672b443a1563a6a1590990a1858c272891338f1e98dde6043 WatchSource:0}: Error finding container 96494011e81a653672b443a1563a6a1590990a1858c272891338f1e98dde6043: Status 404 returned error can't find the container with id 96494011e81a653672b443a1563a6a1590990a1858c272891338f1e98dde6043
Jul 07 12:01:08.323866 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:08.323838    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" event={"ID":"f705fc06-56a8-428f-926b-fd2a0b6530a1","Type":"ContainerStarted","Data":"1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de"}
Jul 07 12:01:08.324196 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:08.323890    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" event={"ID":"f705fc06-56a8-428f-926b-fd2a0b6530a1","Type":"ContainerStarted","Data":"027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa"}
Jul 07 12:01:08.324196 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:08.323915    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:08.324196 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:08.323933    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" event={"ID":"f705fc06-56a8-428f-926b-fd2a0b6530a1","Type":"ContainerStarted","Data":"96494011e81a653672b443a1563a6a1590990a1858c272891338f1e98dde6043"}
Jul 07 12:01:08.324196 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:08.323953    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:08.325295 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:08.325272    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 12:01:08.352754 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:08.352709    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podStartSLOduration=2.352695586 podStartE2EDuration="2.352695586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:01:08.352340361 +0000 UTC m=+1254.547917406" watchObservedRunningTime="2026-07-07 12:01:08.352695586 +0000 UTC m=+1254.548272607"
Jul 07 12:01:09.153620 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.153579    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 12:01:09.158862 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.158839    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 12:01:09.327730 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.327690    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 12:01:09.431849 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.431815    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:01:09.540572 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.540551    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 12:01:09.605570 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.605541    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a35811-de8e-4483-b222-f574a9566720-proxy-tls\") pod \"34a35811-de8e-4483-b222-f574a9566720\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") "
Jul 07 12:01:09.605570 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.605574    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftmrf\" (UniqueName: \"kubernetes.io/projected/34a35811-de8e-4483-b222-f574a9566720-kube-api-access-ftmrf\") pod \"34a35811-de8e-4483-b222-f574a9566720\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") "
Jul 07 12:01:09.605774 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.605677    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a35811-de8e-4483-b222-f574a9566720-kube-rbac-proxy-sar-config\") pod \"34a35811-de8e-4483-b222-f574a9566720\" (UID: \"34a35811-de8e-4483-b222-f574a9566720\") "
Jul 07 12:01:09.605992 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.605965    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a35811-de8e-4483-b222-f574a9566720-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34a35811-de8e-4483-b222-f574a9566720" (UID: "34a35811-de8e-4483-b222-f574a9566720"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:01:09.607527 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.607507    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a35811-de8e-4483-b222-f574a9566720-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34a35811-de8e-4483-b222-f574a9566720" (UID: "34a35811-de8e-4483-b222-f574a9566720"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:01:09.607627 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.607559    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a35811-de8e-4483-b222-f574a9566720-kube-api-access-ftmrf" (OuterVolumeSpecName: "kube-api-access-ftmrf") pod "34a35811-de8e-4483-b222-f574a9566720" (UID: "34a35811-de8e-4483-b222-f574a9566720"). InnerVolumeSpecName "kube-api-access-ftmrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:01:09.706590 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.706537    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a35811-de8e-4483-b222-f574a9566720-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:01:09.706590 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.706559    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftmrf\" (UniqueName: \"kubernetes.io/projected/34a35811-de8e-4483-b222-f574a9566720-kube-api-access-ftmrf\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:01:09.706590 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:09.706570    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a35811-de8e-4483-b222-f574a9566720-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:01:10.332940 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.332907    2572 generic.go:356] "Generic (PLEG): container finished" podID="34a35811-de8e-4483-b222-f574a9566720" containerID="8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf" exitCode=0
Jul 07 12:01:10.333306 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.332970    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"
Jul 07 12:01:10.333306 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.332979    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" event={"ID":"34a35811-de8e-4483-b222-f574a9566720","Type":"ContainerDied","Data":"8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf"}
Jul 07 12:01:10.333306 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.333049    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn" event={"ID":"34a35811-de8e-4483-b222-f574a9566720","Type":"ContainerDied","Data":"e944917093c50685e2a45dc6c15626f96af9e6dc71e96d61556a8017c3919458"}
Jul 07 12:01:10.333306 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.333075    2572 scope.go:117] "RemoveContainer" containerID="c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398"
Jul 07 12:01:10.341512 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.341497    2572 scope.go:117] "RemoveContainer" containerID="8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf"
Jul 07 12:01:10.350397 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.348931    2572 scope.go:117] "RemoveContainer" containerID="c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398"
Jul 07 12:01:10.350799 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:01:10.350780    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398\": container with ID starting with c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398 not found: ID does not exist" containerID="c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398"
Jul 07 12:01:10.350864 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.350807    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398"} err="failed to get container status \"c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398\": rpc error: code = NotFound desc = could not find container \"c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398\": container with ID starting with c9d28873973bf6dafceea769cb67c110c25f06af9c83e4a0368b3e0fc3bcc398 not found: ID does not exist"
Jul 07 12:01:10.350864 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.350824    2572 scope.go:117] "RemoveContainer" containerID="8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf"
Jul 07 12:01:10.351047 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:01:10.351032    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf\": container with ID starting with 8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf not found: ID does not exist" containerID="8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf"
Jul 07 12:01:10.351090 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.351051    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf"} err="failed to get container status \"8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf\": rpc error: code = NotFound desc = could not find container \"8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf\": container with ID starting with 8dab97e858a6a080509c54d8a43a52247128c349ab7142019d9db00e29cc10bf not found: ID does not exist"
Jul 07 12:01:10.353559 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.353540    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"]
Jul 07 12:01:10.360721 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.360699    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn"]
Jul 07 12:01:10.439150 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:10.439131    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34a35811-de8e-4483-b222-f574a9566720" path="/var/lib/kubelet/pods/34a35811-de8e-4483-b222-f574a9566720/volumes"
Jul 07 12:01:14.333338 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:14.333308    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:14.333864 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:14.333836    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 12:01:14.431787 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:14.431759    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:01:19.431329 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:19.431288    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:01:19.431770 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:19.431409    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 12:01:24.334623 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:24.334587    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 12:01:24.432598 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:24.432556    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:01:29.431780 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:29.431736    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:01:34.334280 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:34.334242    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 12:01:34.431636 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:34.431607    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:01:36.440942 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:36.440912    2572 generic.go:356] "Generic (PLEG): container finished" podID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerID="326f3b2414b305b213e354dbe2a59a6c702d339e58dd4a254578ce623c1f6066" exitCode=0
Jul 07 12:01:36.441299 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:36.440978    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" event={"ID":"8235a072-409a-495b-bd9e-0a35ef4c27e0","Type":"ContainerDied","Data":"326f3b2414b305b213e354dbe2a59a6c702d339e58dd4a254578ce623c1f6066"}
Jul 07 12:01:36.842723 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:36.842701    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 12:01:36.992265 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:36.992206    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8235a072-409a-495b-bd9e-0a35ef4c27e0-openshift-service-ca-bundle\") pod \"8235a072-409a-495b-bd9e-0a35ef4c27e0\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") "
Jul 07 12:01:36.992454 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:36.992269    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls\") pod \"8235a072-409a-495b-bd9e-0a35ef4c27e0\" (UID: \"8235a072-409a-495b-bd9e-0a35ef4c27e0\") "
Jul 07 12:01:36.992591 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:36.992565    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8235a072-409a-495b-bd9e-0a35ef4c27e0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8235a072-409a-495b-bd9e-0a35ef4c27e0" (UID: "8235a072-409a-495b-bd9e-0a35ef4c27e0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:01:36.994172 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:36.994153    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8235a072-409a-495b-bd9e-0a35ef4c27e0" (UID: "8235a072-409a-495b-bd9e-0a35ef4c27e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:01:37.092862 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:37.092842    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8235a072-409a-495b-bd9e-0a35ef4c27e0-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:01:37.092862 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:37.092862    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8235a072-409a-495b-bd9e-0a35ef4c27e0-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:01:37.446088 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:37.446061    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"
Jul 07 12:01:37.446516 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:37.446063    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts" event={"ID":"8235a072-409a-495b-bd9e-0a35ef4c27e0","Type":"ContainerDied","Data":"4b750a8d2ade4fc5ecd027fa510b98ab6d3577f2fe572adb2480e01f2b9e3f40"}
Jul 07 12:01:37.446516 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:37.446179    2572 scope.go:117] "RemoveContainer" containerID="326f3b2414b305b213e354dbe2a59a6c702d339e58dd4a254578ce623c1f6066"
Jul 07 12:01:37.471714 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:37.471692    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"]
Jul 07 12:01:37.475307 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:37.475284    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts"]
Jul 07 12:01:38.438472 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:38.438436    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" path="/var/lib/kubelet/pods/8235a072-409a-495b-bd9e-0a35ef4c27e0/volumes"
Jul 07 12:01:44.334700 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:44.334644    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 12:01:54.334421 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:54.334392    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:01:56.081271 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.081242    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"]
Jul 07 12:01:56.081634 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.081438    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43" containerID="cri-o://856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5" gracePeriod=30
Jul 07 12:01:56.105852 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.105825    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:56.240868 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.240836    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"]
Jul 07 12:01:56.241140 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.241106    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container" containerID="cri-o://fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed" gracePeriod=30
Jul 07 12:01:56.241270 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.241118    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kube-rbac-proxy" containerID="cri-o://dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67" gracePeriod=30
Jul 07 12:01:56.249376 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.249351    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:56.258750 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.258729    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"]
Jul 07 12:01:56.259137 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259122    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container"
Jul 07 12:01:56.259184 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259139    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container"
Jul 07 12:01:56.259184 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259151    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2"
Jul 07 12:01:56.259184 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259156    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2"
Jul 07 12:01:56.259184 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259167    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kube-rbac-proxy"
Jul 07 12:01:56.259184 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259173    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kube-rbac-proxy"
Jul 07 12:01:56.259337 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259221    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8235a072-409a-495b-bd9e-0a35ef4c27e0" containerName="switch-graph-5e1d2"
Jul 07 12:01:56.259337 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259232    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kserve-container"
Jul 07 12:01:56.259337 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.259239    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a35811-de8e-4483-b222-f574a9566720" containerName="kube-rbac-proxy"
Jul 07 12:01:56.261893 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.261877    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.264176 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.264158    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a4899-predictor-serving-cert\""
Jul 07 12:01:56.264288 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.264163    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a4899-kube-rbac-proxy-sar-config\""
Jul 07 12:01:56.274520 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.274500    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"]
Jul 07 12:01:56.383929 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.383898    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:56.424427 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.424401    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h56fv\" (UniqueName: \"kubernetes.io/projected/b733aee6-0587-4995-959d-8b63b408156c-kube-api-access-h56fv\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.424557 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.424437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b733aee6-0587-4995-959d-8b63b408156c-proxy-tls\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.424557 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.424464    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b733aee6-0587-4995-959d-8b63b408156c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.482058 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.482029    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:56.516708 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.516683    2572 generic.go:356] "Generic (PLEG): container finished" podID="1156089a-302d-410d-a777-37f2860a1f95" containerID="dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67" exitCode=2
Jul 07 12:01:56.516805 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.516756    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" event={"ID":"1156089a-302d-410d-a777-37f2860a1f95","Type":"ContainerDied","Data":"dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67"}
Jul 07 12:01:56.525002 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.524981    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b733aee6-0587-4995-959d-8b63b408156c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.525098 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.525075    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h56fv\" (UniqueName: \"kubernetes.io/projected/b733aee6-0587-4995-959d-8b63b408156c-kube-api-access-h56fv\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.525159 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.525110    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b733aee6-0587-4995-959d-8b63b408156c-proxy-tls\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.525612 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.525594    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b733aee6-0587-4995-959d-8b63b408156c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.527430 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.527410    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b733aee6-0587-4995-959d-8b63b408156c-proxy-tls\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.532795 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.532778    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h56fv\" (UniqueName: \"kubernetes.io/projected/b733aee6-0587-4995-959d-8b63b408156c-kube-api-access-h56fv\") pod \"success-200-isvc-a4899-predictor-65f55d68f9-f47hd\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") " pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.571605 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.571586    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:56.592784 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.592755    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:01:56.708250 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:56.708227    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"]
Jul 07 12:01:56.710202 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:01:56.710178    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb733aee6_0587_4995_959d_8b63b408156c.slice/crio-123dab1f9e9867b8ceb3f359b6243d4a8e69ee9a2e277d65ea86fe0091edcee4 WatchSource:0}: Error finding container 123dab1f9e9867b8ceb3f359b6243d4a8e69ee9a2e277d65ea86fe0091edcee4: Status 404 returned error can't find the container with id 123dab1f9e9867b8ceb3f359b6243d4a8e69ee9a2e277d65ea86fe0091edcee4
Jul 07 12:01:57.522848 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:57.522805    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" event={"ID":"b733aee6-0587-4995-959d-8b63b408156c","Type":"ContainerStarted","Data":"52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf"}
Jul 07 12:01:57.523434 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:57.522905    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" event={"ID":"b733aee6-0587-4995-959d-8b63b408156c","Type":"ContainerStarted","Data":"6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa"}
Jul 07 12:01:57.523434 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:57.522947    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:57.523434 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:57.522987    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:01:57.523434 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:57.523016    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" event={"ID":"b733aee6-0587-4995-959d-8b63b408156c","Type":"ContainerStarted","Data":"123dab1f9e9867b8ceb3f359b6243d4a8e69ee9a2e277d65ea86fe0091edcee4"}
Jul 07 12:01:57.524339 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:57.524314    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:01:57.542338 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:57.542294    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podStartSLOduration=1.542279613 podStartE2EDuration="1.542279613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:01:57.540978525 +0000 UTC m=+1303.736555548" watchObservedRunningTime="2026-07-07 12:01:57.542279613 +0000 UTC m=+1303.737856634"
Jul 07 12:01:58.372500 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:58.372462    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 07 12:01:58.377485 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:58.377461    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 12:01:58.526247 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:58.526210    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:01:58.616541 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:58.616505    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:01:58.883297 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:58.883273    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 12:01:59.043275 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.043246    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1156089a-302d-410d-a777-37f2860a1f95-proxy-tls\") pod \"1156089a-302d-410d-a777-37f2860a1f95\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") "
Jul 07 12:01:59.043419 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.043330    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1156089a-302d-410d-a777-37f2860a1f95-kube-rbac-proxy-sar-config\") pod \"1156089a-302d-410d-a777-37f2860a1f95\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") "
Jul 07 12:01:59.043419 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.043363    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7psn\" (UniqueName: \"kubernetes.io/projected/1156089a-302d-410d-a777-37f2860a1f95-kube-api-access-g7psn\") pod \"1156089a-302d-410d-a777-37f2860a1f95\" (UID: \"1156089a-302d-410d-a777-37f2860a1f95\") "
Jul 07 12:01:59.043699 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.043675    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1156089a-302d-410d-a777-37f2860a1f95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1156089a-302d-410d-a777-37f2860a1f95" (UID: "1156089a-302d-410d-a777-37f2860a1f95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:01:59.045306 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.045284    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1156089a-302d-410d-a777-37f2860a1f95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1156089a-302d-410d-a777-37f2860a1f95" (UID: "1156089a-302d-410d-a777-37f2860a1f95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:01:59.045306 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.045290    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1156089a-302d-410d-a777-37f2860a1f95-kube-api-access-g7psn" (OuterVolumeSpecName: "kube-api-access-g7psn") pod "1156089a-302d-410d-a777-37f2860a1f95" (UID: "1156089a-302d-410d-a777-37f2860a1f95"). InnerVolumeSpecName "kube-api-access-g7psn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:01:59.144582 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.144560    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1156089a-302d-410d-a777-37f2860a1f95-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:01:59.144582 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.144579    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1156089a-302d-410d-a777-37f2860a1f95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:01:59.144733 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.144589    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7psn\" (UniqueName: \"kubernetes.io/projected/1156089a-302d-410d-a777-37f2860a1f95-kube-api-access-g7psn\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:01:59.531673 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.531631    2572 generic.go:356] "Generic (PLEG): container finished" podID="1156089a-302d-410d-a777-37f2860a1f95" containerID="fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed" exitCode=0
Jul 07 12:01:59.532103 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.531679    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" event={"ID":"1156089a-302d-410d-a777-37f2860a1f95","Type":"ContainerDied","Data":"fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed"}
Jul 07 12:01:59.532103 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.531697    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"
Jul 07 12:01:59.532103 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.531727    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st" event={"ID":"1156089a-302d-410d-a777-37f2860a1f95","Type":"ContainerDied","Data":"55262f903d3751eb2201b17d1f1e4101b190764bb4315f7bb41856573cf8b7a0"}
Jul 07 12:01:59.532103 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.531754    2572 scope.go:117] "RemoveContainer" containerID="dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67"
Jul 07 12:01:59.540818 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.540803    2572 scope.go:117] "RemoveContainer" containerID="fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed"
Jul 07 12:01:59.547620 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.547605    2572 scope.go:117] "RemoveContainer" containerID="dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67"
Jul 07 12:01:59.547880 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:01:59.547863    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67\": container with ID starting with dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67 not found: ID does not exist" containerID="dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67"
Jul 07 12:01:59.547960 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.547893    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67"} err="failed to get container status \"dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67\": rpc error: code = NotFound desc = could not find container \"dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67\": container with ID starting with dc9f7c12adf77b28cafe7583a561df34be9870e865f66a99b12dcbe239c07e67 not found: ID does not exist"
Jul 07 12:01:59.547960 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.547917    2572 scope.go:117] "RemoveContainer" containerID="fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed"
Jul 07 12:01:59.548136 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:01:59.548115    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed\": container with ID starting with fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed not found: ID does not exist" containerID="fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed"
Jul 07 12:01:59.548177 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.548142    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed"} err="failed to get container status \"fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed\": rpc error: code = NotFound desc = could not find container \"fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed\": container with ID starting with fed9ecbaa1a9309171ee48cfbef3c3780991e06572c01223b1582e92b22fe2ed not found: ID does not exist"
Jul 07 12:01:59.553239 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.553217    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"]
Jul 07 12:01:59.556976 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:01:59.556959    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st"]
Jul 07 12:02:00.439288 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:00.439257    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1156089a-302d-410d-a777-37f2860a1f95" path="/var/lib/kubelet/pods/1156089a-302d-410d-a777-37f2860a1f95/volumes"
Jul 07 12:02:01.003230 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:01.003201    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:02:03.530933 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:03.530905    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:02:03.531391 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:03.531361    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:03.616610 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:03.616579    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:06.567461 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.567426    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"]
Jul 07 12:02:06.567884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.567726    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container"
Jul 07 12:02:06.567884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.567737    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container"
Jul 07 12:02:06.567884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.567746    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kube-rbac-proxy"
Jul 07 12:02:06.567884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.567752    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kube-rbac-proxy"
Jul 07 12:02:06.567884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.567808    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kube-rbac-proxy"
Jul 07 12:02:06.567884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.567818    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1156089a-302d-410d-a777-37f2860a1f95" containerName="kserve-container"
Jul 07 12:02:06.572097 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.572077    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:06.575325 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.575307    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-462dc-serving-cert\""
Jul 07 12:02:06.575861 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.575838    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-462dc-kube-rbac-proxy-sar-config\""
Jul 07 12:02:06.578204 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.578185    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:02:06.586241 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.586224    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"]
Jul 07 12:02:06.692384 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.692362    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14b2cf8-e9aa-4211-ad75-d69be586de04-openshift-service-ca-bundle\") pod \"ensemble-graph-462dc-867dff89-xr95l\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") " pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:06.692483 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.692404    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls\") pod \"ensemble-graph-462dc-867dff89-xr95l\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") " pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:06.792946 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.792923    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls\") pod \"ensemble-graph-462dc-867dff89-xr95l\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") " pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:06.793022 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.792978    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14b2cf8-e9aa-4211-ad75-d69be586de04-openshift-service-ca-bundle\") pod \"ensemble-graph-462dc-867dff89-xr95l\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") " pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:06.793074 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:02:06.793059    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-462dc-serving-cert: secret "ensemble-graph-462dc-serving-cert" not found
Jul 07 12:02:06.793124 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:02:06.793113    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls podName:f14b2cf8-e9aa-4211-ad75-d69be586de04 nodeName:}" failed. No retries permitted until 2026-07-07 12:02:07.29309575 +0000 UTC m=+1313.488672754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls") pod "ensemble-graph-462dc-867dff89-xr95l" (UID: "f14b2cf8-e9aa-4211-ad75-d69be586de04") : secret "ensemble-graph-462dc-serving-cert" not found
Jul 07 12:02:06.793476 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.793461    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14b2cf8-e9aa-4211-ad75-d69be586de04-openshift-service-ca-bundle\") pod \"ensemble-graph-462dc-867dff89-xr95l\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") " pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:06.959245 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:06.959175    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:02:07.296610 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:07.296580    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls\") pod \"ensemble-graph-462dc-867dff89-xr95l\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") " pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:07.298799 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:07.298779    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls\") pod \"ensemble-graph-462dc-867dff89-xr95l\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") " pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:07.482249 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:07.482217    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:07.631078 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:07.631038    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"]
Jul 07 12:02:07.634917 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:02:07.634888    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14b2cf8_e9aa_4211_ad75_d69be586de04.slice/crio-99e4dd6d5664e6d36190d902dabcf2a4fa7000b63e342f03631b30cca32d832a WatchSource:0}: Error finding container 99e4dd6d5664e6d36190d902dabcf2a4fa7000b63e342f03631b30cca32d832a: Status 404 returned error can't find the container with id 99e4dd6d5664e6d36190d902dabcf2a4fa7000b63e342f03631b30cca32d832a
Jul 07 12:02:08.570950 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:08.570908    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" event={"ID":"f14b2cf8-e9aa-4211-ad75-d69be586de04","Type":"ContainerStarted","Data":"2027d951d16074766f457f5c4a316c8e684c8596894cbcd67064d89709fed400"}
Jul 07 12:02:08.571103 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:08.570968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" event={"ID":"f14b2cf8-e9aa-4211-ad75-d69be586de04","Type":"ContainerStarted","Data":"99e4dd6d5664e6d36190d902dabcf2a4fa7000b63e342f03631b30cca32d832a"}
Jul 07 12:02:08.571103 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:08.571026    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:08.591876 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:08.591831    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" podStartSLOduration=2.591817458 podStartE2EDuration="2.591817458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:02:08.589862428 +0000 UTC m=+1314.785439452" watchObservedRunningTime="2026-07-07 12:02:08.591817458 +0000 UTC m=+1314.787394517"
Jul 07 12:02:08.616620 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:08.616595    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:08.616719 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:08.616707    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 12:02:13.532003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:13.531967    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:13.616961 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:13.616928    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:14.581184 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:14.581153    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:16.656809 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:16.656782    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"]
Jul 07 12:02:16.657206 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:16.656954    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc" containerID="cri-o://2027d951d16074766f457f5c4a316c8e684c8596894cbcd67064d89709fed400" gracePeriod=30
Jul 07 12:02:16.718447 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:16.718422    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:02:16.906327 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:16.906296    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"]
Jul 07 12:02:16.911102 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:16.911042    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:16.916620 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:16.916592    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e7782-predictor-serving-cert\""
Jul 07 12:02:16.916765 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:16.916748    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e7782-kube-rbac-proxy-sar-config\""
Jul 07 12:02:16.924737 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:16.924716    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"]
Jul 07 12:02:17.040114 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.040082    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"]
Jul 07 12:02:17.040318 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.040294    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container" containerID="cri-o://027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa" gracePeriod=30
Jul 07 12:02:17.040369 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.040329    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kube-rbac-proxy" containerID="cri-o://1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de" gracePeriod=30
Jul 07 12:02:17.065754 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.065724    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-proxy-tls\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.065872 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.065829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.065961 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.065939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tfm\" (UniqueName: \"kubernetes.io/projected/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-api-access-28tfm\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.167044 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.166992    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tfm\" (UniqueName: \"kubernetes.io/projected/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-api-access-28tfm\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.167161 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.167039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-proxy-tls\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.167161 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.167087    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.167729 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.167709    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.169282 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.169259    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-proxy-tls\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.186238 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.186218    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tfm\" (UniqueName: \"kubernetes.io/projected/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-api-access-28tfm\") pod \"success-200-isvc-e7782-predictor-74fd9b479-lhdr2\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") " pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.220703 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.220681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.398146 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.398125    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"]
Jul 07 12:02:17.400147 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:02:17.400120    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6142e9d_7af0_433f_b5cb_2aee7ceadacb.slice/crio-9b453481a1296c995d07c32941b3a1e802e03a0120cb69f39c25aae7c5275507 WatchSource:0}: Error finding container 9b453481a1296c995d07c32941b3a1e802e03a0120cb69f39c25aae7c5275507: Status 404 returned error can't find the container with id 9b453481a1296c995d07c32941b3a1e802e03a0120cb69f39c25aae7c5275507
Jul 07 12:02:17.608461 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.608431    2572 generic.go:356] "Generic (PLEG): container finished" podID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerID="1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de" exitCode=2
Jul 07 12:02:17.608586 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.608504    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" event={"ID":"f705fc06-56a8-428f-926b-fd2a0b6530a1","Type":"ContainerDied","Data":"1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de"}
Jul 07 12:02:17.610140 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.610123    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" event={"ID":"f6142e9d-7af0-433f-b5cb-2aee7ceadacb","Type":"ContainerStarted","Data":"d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b"}
Jul 07 12:02:17.610200 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.610166    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" event={"ID":"f6142e9d-7af0-433f-b5cb-2aee7ceadacb","Type":"ContainerStarted","Data":"ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be"}
Jul 07 12:02:17.610200 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.610197    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" event={"ID":"f6142e9d-7af0-433f-b5cb-2aee7ceadacb","Type":"ContainerStarted","Data":"9b453481a1296c995d07c32941b3a1e802e03a0120cb69f39c25aae7c5275507"}
Jul 07 12:02:17.610295 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.610222    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.610404 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.610384    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:17.611705 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.611681    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 12:02:17.643628 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.643580    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podStartSLOduration=1.643563978 podStartE2EDuration="1.643563978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:02:17.643302612 +0000 UTC m=+1323.838879638" watchObservedRunningTime="2026-07-07 12:02:17.643563978 +0000 UTC m=+1323.839141003"
Jul 07 12:02:17.819363 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:17.819338    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:02:18.614476 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:18.614430    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 12:02:18.616882 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:18.616850    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:19.328221 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.328186    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 12:02:19.578313 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.578228    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:19.874526 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.874503    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:02:19.885451 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.885430    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csfd9\" (UniqueName: \"kubernetes.io/projected/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-api-access-csfd9\") pod \"f705fc06-56a8-428f-926b-fd2a0b6530a1\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") "
Jul 07 12:02:19.885558 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.885467    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-rbac-proxy-sar-config\") pod \"f705fc06-56a8-428f-926b-fd2a0b6530a1\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") "
Jul 07 12:02:19.885558 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.885496    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls\") pod \"f705fc06-56a8-428f-926b-fd2a0b6530a1\" (UID: \"f705fc06-56a8-428f-926b-fd2a0b6530a1\") "
Jul 07 12:02:19.885793 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.885770    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f705fc06-56a8-428f-926b-fd2a0b6530a1" (UID: "f705fc06-56a8-428f-926b-fd2a0b6530a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:19.887282 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.887260    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-api-access-csfd9" (OuterVolumeSpecName: "kube-api-access-csfd9") pod "f705fc06-56a8-428f-926b-fd2a0b6530a1" (UID: "f705fc06-56a8-428f-926b-fd2a0b6530a1"). InnerVolumeSpecName "kube-api-access-csfd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:02:19.887345 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.887331    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f705fc06-56a8-428f-926b-fd2a0b6530a1" (UID: "f705fc06-56a8-428f-926b-fd2a0b6530a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:02:19.986058 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.986028    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfd9\" (UniqueName: \"kubernetes.io/projected/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-api-access-csfd9\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:02:19.986058 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.986055    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f705fc06-56a8-428f-926b-fd2a0b6530a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:02:19.986193 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:19.986066    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f705fc06-56a8-428f-926b-fd2a0b6530a1-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:02:20.623344 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.623319    2572 generic.go:356] "Generic (PLEG): container finished" podID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerID="027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa" exitCode=0
Jul 07 12:02:20.623728 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.623363    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"
Jul 07 12:02:20.623728 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.623393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" event={"ID":"f705fc06-56a8-428f-926b-fd2a0b6530a1","Type":"ContainerDied","Data":"027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa"}
Jul 07 12:02:20.623728 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.623447    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm" event={"ID":"f705fc06-56a8-428f-926b-fd2a0b6530a1","Type":"ContainerDied","Data":"96494011e81a653672b443a1563a6a1590990a1858c272891338f1e98dde6043"}
Jul 07 12:02:20.623728 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.623472    2572 scope.go:117] "RemoveContainer" containerID="1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de"
Jul 07 12:02:20.632003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.631985    2572 scope.go:117] "RemoveContainer" containerID="027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa"
Jul 07 12:02:20.639381 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.639356    2572 scope.go:117] "RemoveContainer" containerID="1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de"
Jul 07 12:02:20.639670 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:02:20.639637    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de\": container with ID starting with 1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de not found: ID does not exist" containerID="1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de"
Jul 07 12:02:20.639738 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.639708    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de"} err="failed to get container status \"1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de\": rpc error: code = NotFound desc = could not find container \"1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de\": container with ID starting with 1abe2d796f7cb924b308433795de0af9653bc923542ae28c53d506d96530b8de not found: ID does not exist"
Jul 07 12:02:20.639738 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.639735    2572 scope.go:117] "RemoveContainer" containerID="027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa"
Jul 07 12:02:20.639972 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:02:20.639952    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa\": container with ID starting with 027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa not found: ID does not exist" containerID="027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa"
Jul 07 12:02:20.640031 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.639981    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa"} err="failed to get container status \"027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa\": rpc error: code = NotFound desc = could not find container \"027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa\": container with ID starting with 027a9ac50a67a5a24d627743e2dca0b62a0a14bc56a1be7e96120a1d83e0aaaa not found: ID does not exist"
Jul 07 12:02:20.646202 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.646182    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"]
Jul 07 12:02:20.650943 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:20.650923    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm"]
Jul 07 12:02:22.439489 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:22.439456    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" path="/var/lib/kubelet/pods/f705fc06-56a8-428f-926b-fd2a0b6530a1/volumes"
Jul 07 12:02:23.531410 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:23.531368    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:23.616574 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:23.616546    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:23.619491 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:23.619473    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:02:23.619929 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:23.619905    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 12:02:24.578351 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:24.578317    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:26.218486 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.218463    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 12:02:26.220446 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.220429    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cdc2570-8db9-42ef-a873-cafecd654cc0-openshift-service-ca-bundle\") pod \"1cdc2570-8db9-42ef-a873-cafecd654cc0\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") "
Jul 07 12:02:26.220499 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.220483    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls\") pod \"1cdc2570-8db9-42ef-a873-cafecd654cc0\" (UID: \"1cdc2570-8db9-42ef-a873-cafecd654cc0\") "
Jul 07 12:02:26.220733 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.220712    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cdc2570-8db9-42ef-a873-cafecd654cc0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1cdc2570-8db9-42ef-a873-cafecd654cc0" (UID: "1cdc2570-8db9-42ef-a873-cafecd654cc0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:26.222355 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.222327    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1cdc2570-8db9-42ef-a873-cafecd654cc0" (UID: "1cdc2570-8db9-42ef-a873-cafecd654cc0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:02:26.321649 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.321626    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cdc2570-8db9-42ef-a873-cafecd654cc0-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:02:26.321649 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.321650    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cdc2570-8db9-42ef-a873-cafecd654cc0-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:02:26.648062 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.648033    2572 generic.go:356] "Generic (PLEG): container finished" podID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerID="856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5" exitCode=0
Jul 07 12:02:26.648185 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.648074    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"
Jul 07 12:02:26.648185 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.648118    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" event={"ID":"1cdc2570-8db9-42ef-a873-cafecd654cc0","Type":"ContainerDied","Data":"856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5"}
Jul 07 12:02:26.648185 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.648167    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476" event={"ID":"1cdc2570-8db9-42ef-a873-cafecd654cc0","Type":"ContainerDied","Data":"cd0da7aa5f7b56700ce18aef632f25fdb57b27ae50b794569aa55c54921b2cfc"}
Jul 07 12:02:26.648338 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.648189    2572 scope.go:117] "RemoveContainer" containerID="856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5"
Jul 07 12:02:26.655931 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.655912    2572 scope.go:117] "RemoveContainer" containerID="856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5"
Jul 07 12:02:26.656167 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:02:26.656147    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5\": container with ID starting with 856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5 not found: ID does not exist" containerID="856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5"
Jul 07 12:02:26.656223 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.656176    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5"} err="failed to get container status \"856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5\": rpc error: code = NotFound desc = could not find container \"856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5\": container with ID starting with 856e79a3bfed5330b2f5f8d6a2d92239f0541e5c29da562cee816f3e08d909a5 not found: ID does not exist"
Jul 07 12:02:26.675576 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.675552    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"]
Jul 07 12:02:26.691758 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:26.691736    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476"]
Jul 07 12:02:28.438405 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:28.438360    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" path="/var/lib/kubelet/pods/1cdc2570-8db9-42ef-a873-cafecd654cc0/volumes"
Jul 07 12:02:29.578713 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:29.578680    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:29.579154 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:29.578789    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:33.124086 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:33.124056    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:02:33.532306 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:33.532263    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:33.620748 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:33.620712    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 12:02:34.578342 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:34.578299    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:39.579191 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:39.579113    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:43.531789 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:43.531765    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:02:43.619919 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:43.619888    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 12:02:44.578145 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:44.578089    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:02:46.725477 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.725444    2572 generic.go:356] "Generic (PLEG): container finished" podID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerID="2027d951d16074766f457f5c4a316c8e684c8596894cbcd67064d89709fed400" exitCode=0
Jul 07 12:02:46.725824 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.725518    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" event={"ID":"f14b2cf8-e9aa-4211-ad75-d69be586de04","Type":"ContainerDied","Data":"2027d951d16074766f457f5c4a316c8e684c8596894cbcd67064d89709fed400"}
Jul 07 12:02:46.801898 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.801876    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:46.864402 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.864373    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls\") pod \"f14b2cf8-e9aa-4211-ad75-d69be586de04\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") "
Jul 07 12:02:46.864513 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.864469    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14b2cf8-e9aa-4211-ad75-d69be586de04-openshift-service-ca-bundle\") pod \"f14b2cf8-e9aa-4211-ad75-d69be586de04\" (UID: \"f14b2cf8-e9aa-4211-ad75-d69be586de04\") "
Jul 07 12:02:46.864813 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.864789    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14b2cf8-e9aa-4211-ad75-d69be586de04-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f14b2cf8-e9aa-4211-ad75-d69be586de04" (UID: "f14b2cf8-e9aa-4211-ad75-d69be586de04"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:46.866304 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.866287    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f14b2cf8-e9aa-4211-ad75-d69be586de04" (UID: "f14b2cf8-e9aa-4211-ad75-d69be586de04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:02:46.965492 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.965438    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14b2cf8-e9aa-4211-ad75-d69be586de04-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:02:46.965492 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:46.965458    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14b2cf8-e9aa-4211-ad75-d69be586de04-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:02:47.730801 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:47.730773    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"
Jul 07 12:02:47.731216 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:47.730810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l" event={"ID":"f14b2cf8-e9aa-4211-ad75-d69be586de04","Type":"ContainerDied","Data":"99e4dd6d5664e6d36190d902dabcf2a4fa7000b63e342f03631b30cca32d832a"}
Jul 07 12:02:47.731216 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:47.730869    2572 scope.go:117] "RemoveContainer" containerID="2027d951d16074766f457f5c4a316c8e684c8596894cbcd67064d89709fed400"
Jul 07 12:02:47.753039 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:47.753017    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"]
Jul 07 12:02:47.756636 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:47.756611    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l"]
Jul 07 12:02:48.438620 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:48.438586    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" path="/var/lib/kubelet/pods/f14b2cf8-e9aa-4211-ad75-d69be586de04/volumes"
Jul 07 12:02:53.620415 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:53.620382    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 12:02:56.381931 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.381895    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"]
Jul 07 12:02:56.382367 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382352    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc"
Jul 07 12:02:56.382427 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382371    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc"
Jul 07 12:02:56.382427 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382383    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container"
Jul 07 12:02:56.382427 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382392    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container"
Jul 07 12:02:56.382427 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382410    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kube-rbac-proxy"
Jul 07 12:02:56.382427 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382419    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kube-rbac-proxy"
Jul 07 12:02:56.382705 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382436    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43"
Jul 07 12:02:56.382705 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382445    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43"
Jul 07 12:02:56.382705 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382514    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kserve-container"
Jul 07 12:02:56.382705 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382526    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f14b2cf8-e9aa-4211-ad75-d69be586de04" containerName="ensemble-graph-462dc"
Jul 07 12:02:56.382705 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382537    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdc2570-8db9-42ef-a873-cafecd654cc0" containerName="sequence-graph-07b43"
Jul 07 12:02:56.382705 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.382548    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f705fc06-56a8-428f-926b-fd2a0b6530a1" containerName="kube-rbac-proxy"
Jul 07 12:02:56.387007 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.386985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:56.391017 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.391000    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a4899-kube-rbac-proxy-sar-config\""
Jul 07 12:02:56.391017 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.391007    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a4899-serving-cert\""
Jul 07 12:02:56.398301 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.398279    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:02:56.398583 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.398565    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"]
Jul 07 12:02:56.433605 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.433584    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c59f328-1ff7-4e5a-8d2a-20f644472158-proxy-tls\") pod \"sequence-graph-a4899-7dd75cd587-46gmq\" (UID: \"6c59f328-1ff7-4e5a-8d2a-20f644472158\") " pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:56.433717 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.433646    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c59f328-1ff7-4e5a-8d2a-20f644472158-openshift-service-ca-bundle\") pod \"sequence-graph-a4899-7dd75cd587-46gmq\" (UID: \"6c59f328-1ff7-4e5a-8d2a-20f644472158\") " pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:56.535017 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.534975    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c59f328-1ff7-4e5a-8d2a-20f644472158-openshift-service-ca-bundle\") pod \"sequence-graph-a4899-7dd75cd587-46gmq\" (UID: \"6c59f328-1ff7-4e5a-8d2a-20f644472158\") " pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:56.535110 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.535050    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c59f328-1ff7-4e5a-8d2a-20f644472158-proxy-tls\") pod \"sequence-graph-a4899-7dd75cd587-46gmq\" (UID: \"6c59f328-1ff7-4e5a-8d2a-20f644472158\") " pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:56.535530 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.535508    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c59f328-1ff7-4e5a-8d2a-20f644472158-openshift-service-ca-bundle\") pod \"sequence-graph-a4899-7dd75cd587-46gmq\" (UID: \"6c59f328-1ff7-4e5a-8d2a-20f644472158\") " pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:56.537587 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.537563    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c59f328-1ff7-4e5a-8d2a-20f644472158-proxy-tls\") pod \"sequence-graph-a4899-7dd75cd587-46gmq\" (UID: \"6c59f328-1ff7-4e5a-8d2a-20f644472158\") " pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:56.577795 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.577771    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:02:56.697403 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.697343    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:56.821480 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:56.821447    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"]
Jul 07 12:02:56.824474 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:02:56.824452    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c59f328_1ff7_4e5a_8d2a_20f644472158.slice/crio-6b4eec1762c6907dd165d67031ceaf3c102dc1de1019ad4f4721184d88f8610d WatchSource:0}: Error finding container 6b4eec1762c6907dd165d67031ceaf3c102dc1de1019ad4f4721184d88f8610d: Status 404 returned error can't find the container with id 6b4eec1762c6907dd165d67031ceaf3c102dc1de1019ad4f4721184d88f8610d
Jul 07 12:02:57.770704 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:57.770651    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" event={"ID":"6c59f328-1ff7-4e5a-8d2a-20f644472158","Type":"ContainerStarted","Data":"11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53"}
Jul 07 12:02:57.771113 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:57.770740    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" event={"ID":"6c59f328-1ff7-4e5a-8d2a-20f644472158","Type":"ContainerStarted","Data":"6b4eec1762c6907dd165d67031ceaf3c102dc1de1019ad4f4721184d88f8610d"}
Jul 07 12:02:57.771113 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:57.770839    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:02:57.797196 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:02:57.797154    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" podStartSLOduration=1.7971416059999998 podStartE2EDuration="1.797141606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:02:57.795519582 +0000 UTC m=+1363.991096604" watchObservedRunningTime="2026-07-07 12:02:57.797141606 +0000 UTC m=+1363.992718629"
Jul 07 12:03:03.620835 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:03.620805    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:03:03.781670 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:03.781632    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:03:06.378733 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.378702    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"]
Jul 07 12:03:06.379097 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.378903    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899" containerID="cri-o://11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53" gracePeriod=30
Jul 07 12:03:06.406187 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.406162    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:03:06.504046 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.504018    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"]
Jul 07 12:03:06.504271 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.504225    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container" containerID="cri-o://6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa" gracePeriod=30
Jul 07 12:03:06.504396 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.504249    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kube-rbac-proxy" containerID="cri-o://52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf" gracePeriod=30
Jul 07 12:03:06.528260 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.528238    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:03:06.548417 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.548396    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"]
Jul 07 12:03:06.551641 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.551627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.554377 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.554361    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b491e-predictor-serving-cert\""
Jul 07 12:03:06.555596 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.555577    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b491e-kube-rbac-proxy-sar-config\""
Jul 07 12:03:06.563391 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.563372    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"]
Jul 07 12:03:06.609443 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.609421    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9jk\" (UniqueName: \"kubernetes.io/projected/baf1012b-7087-43b9-b457-481b96490086-kube-api-access-gr9jk\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.609581 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.609484    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.609581 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.609507    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf1012b-7087-43b9-b457-481b96490086-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.629781 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.629732    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:03:06.709884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.709859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9jk\" (UniqueName: \"kubernetes.io/projected/baf1012b-7087-43b9-b457-481b96490086-kube-api-access-gr9jk\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.710024 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.709933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.710024 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.709958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf1012b-7087-43b9-b457-481b96490086-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.710153 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:03:06.710072    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b491e-predictor-serving-cert: secret "success-200-isvc-b491e-predictor-serving-cert" not found
Jul 07 12:03:06.710153 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:03:06.710142    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls podName:baf1012b-7087-43b9-b457-481b96490086 nodeName:}" failed. No retries permitted until 2026-07-07 12:03:07.210123567 +0000 UTC m=+1373.405700567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls") pod "success-200-isvc-b491e-predictor-7559fc655f-hscwb" (UID: "baf1012b-7087-43b9-b457-481b96490086") : secret "success-200-isvc-b491e-predictor-serving-cert" not found
Jul 07 12:03:06.710631 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.710609    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf1012b-7087-43b9-b457-481b96490086-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.710764 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.710739    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:03:06.718572 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.718551    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9jk\" (UniqueName: \"kubernetes.io/projected/baf1012b-7087-43b9-b457-481b96490086-kube-api-access-gr9jk\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:06.782061 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.782033    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:03:06.807330 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.807308    2572 generic.go:356] "Generic (PLEG): container finished" podID="b733aee6-0587-4995-959d-8b63b408156c" containerID="52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf" exitCode=2
Jul 07 12:03:06.807436 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.807338    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" event={"ID":"b733aee6-0587-4995-959d-8b63b408156c","Type":"ContainerDied","Data":"52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf"}
Jul 07 12:03:06.882724 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:06.882638    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:03:07.213717 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.213615    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:07.215926 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.215908    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls\") pod \"success-200-isvc-b491e-predictor-7559fc655f-hscwb\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") " pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:07.461753 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.461719    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:07.582197 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.582174    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"]
Jul 07 12:03:07.584458 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:03:07.584435    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf1012b_7087_43b9_b457_481b96490086.slice/crio-b328912df74fee85847beb78059702fe416b2bcdb2d582d048f04416e4cec43b WatchSource:0}: Error finding container b328912df74fee85847beb78059702fe416b2bcdb2d582d048f04416e4cec43b: Status 404 returned error can't find the container with id b328912df74fee85847beb78059702fe416b2bcdb2d582d048f04416e4cec43b
Jul 07 12:03:07.813264 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.813234    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" event={"ID":"baf1012b-7087-43b9-b457-481b96490086","Type":"ContainerStarted","Data":"5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa"}
Jul 07 12:03:07.813407 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.813285    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" event={"ID":"baf1012b-7087-43b9-b457-481b96490086","Type":"ContainerStarted","Data":"cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef"}
Jul 07 12:03:07.813407 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.813306    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" event={"ID":"baf1012b-7087-43b9-b457-481b96490086","Type":"ContainerStarted","Data":"b328912df74fee85847beb78059702fe416b2bcdb2d582d048f04416e4cec43b"}
Jul 07 12:03:07.813407 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.813330    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:07.833034 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:07.832976    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podStartSLOduration=1.832953659 podStartE2EDuration="1.832953659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:03:07.830998839 +0000 UTC m=+1374.026575874" watchObservedRunningTime="2026-07-07 12:03:07.832953659 +0000 UTC m=+1374.028530685"
Jul 07 12:03:08.526867 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:08.526827    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 07 12:03:08.779878 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:08.779800    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:03:08.817492 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:08.817465    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:08.818640 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:08.818614    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 12:03:09.346461 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.346439    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:03:09.432490 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.432408    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h56fv\" (UniqueName: \"kubernetes.io/projected/b733aee6-0587-4995-959d-8b63b408156c-kube-api-access-h56fv\") pod \"b733aee6-0587-4995-959d-8b63b408156c\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") "
Jul 07 12:03:09.432490 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.432451    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b733aee6-0587-4995-959d-8b63b408156c-proxy-tls\") pod \"b733aee6-0587-4995-959d-8b63b408156c\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") "
Jul 07 12:03:09.432742 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.432581    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b733aee6-0587-4995-959d-8b63b408156c-kube-rbac-proxy-sar-config\") pod \"b733aee6-0587-4995-959d-8b63b408156c\" (UID: \"b733aee6-0587-4995-959d-8b63b408156c\") "
Jul 07 12:03:09.432945 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.432923    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b733aee6-0587-4995-959d-8b63b408156c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b733aee6-0587-4995-959d-8b63b408156c" (UID: "b733aee6-0587-4995-959d-8b63b408156c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:03:09.434470 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.434445    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b733aee6-0587-4995-959d-8b63b408156c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b733aee6-0587-4995-959d-8b63b408156c" (UID: "b733aee6-0587-4995-959d-8b63b408156c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:03:09.434585 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.434499    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b733aee6-0587-4995-959d-8b63b408156c-kube-api-access-h56fv" (OuterVolumeSpecName: "kube-api-access-h56fv") pod "b733aee6-0587-4995-959d-8b63b408156c" (UID: "b733aee6-0587-4995-959d-8b63b408156c"). InnerVolumeSpecName "kube-api-access-h56fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:03:09.533756 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.533725    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b733aee6-0587-4995-959d-8b63b408156c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:03:09.533756 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.533745    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h56fv\" (UniqueName: \"kubernetes.io/projected/b733aee6-0587-4995-959d-8b63b408156c-kube-api-access-h56fv\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:03:09.534192 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.533755    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b733aee6-0587-4995-959d-8b63b408156c-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:03:09.822860 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.822834    2572 generic.go:356] "Generic (PLEG): container finished" podID="b733aee6-0587-4995-959d-8b63b408156c" containerID="6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa" exitCode=0
Jul 07 12:03:09.823015 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.822888    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"
Jul 07 12:03:09.823015 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.822916    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" event={"ID":"b733aee6-0587-4995-959d-8b63b408156c","Type":"ContainerDied","Data":"6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa"}
Jul 07 12:03:09.823015 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.822972    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd" event={"ID":"b733aee6-0587-4995-959d-8b63b408156c","Type":"ContainerDied","Data":"123dab1f9e9867b8ceb3f359b6243d4a8e69ee9a2e277d65ea86fe0091edcee4"}
Jul 07 12:03:09.823015 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.822998    2572 scope.go:117] "RemoveContainer" containerID="52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf"
Jul 07 12:03:09.823405 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.823373    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 12:03:09.831714 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.831697    2572 scope.go:117] "RemoveContainer" containerID="6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa"
Jul 07 12:03:09.839134 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.839118    2572 scope.go:117] "RemoveContainer" containerID="52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf"
Jul 07 12:03:09.839370 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:03:09.839353    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf\": container with ID starting with 52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf not found: ID does not exist" containerID="52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf"
Jul 07 12:03:09.839431 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.839377    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf"} err="failed to get container status \"52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf\": rpc error: code = NotFound desc = could not find container \"52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf\": container with ID starting with 52ebb06b8ff0b8c14ceca303384b8c6bb87ce9f2f2f1ccf96d5d145e6b8a1caf not found: ID does not exist"
Jul 07 12:03:09.839431 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.839393    2572 scope.go:117] "RemoveContainer" containerID="6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa"
Jul 07 12:03:09.839608 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:03:09.839591    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa\": container with ID starting with 6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa not found: ID does not exist" containerID="6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa"
Jul 07 12:03:09.839652 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.839612    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa"} err="failed to get container status \"6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa\": rpc error: code = NotFound desc = could not find container \"6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa\": container with ID starting with 6bc7ec47adeb1b383fedb8b56529330a6e1fff5d4b27fc3cc99ed8909b4a66fa not found: ID does not exist"
Jul 07 12:03:09.844283 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.844168    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"]
Jul 07 12:03:09.845538 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:09.845521    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd"]
Jul 07 12:03:10.438228 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:10.438200    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b733aee6-0587-4995-959d-8b63b408156c" path="/var/lib/kubelet/pods/b733aee6-0587-4995-959d-8b63b408156c/volumes"
Jul 07 12:03:13.779372 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:13.779329    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:03:14.827461 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:14.827437    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:03:14.827884 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:14.827859    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 12:03:16.877372 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.877344    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"]
Jul 07 12:03:16.877770 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.877637    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container"
Jul 07 12:03:16.877770 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.877649    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container"
Jul 07 12:03:16.877770 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.877680    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kube-rbac-proxy"
Jul 07 12:03:16.877770 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.877686    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kube-rbac-proxy"
Jul 07 12:03:16.877770 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.877737    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kserve-container"
Jul 07 12:03:16.877770 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.877744    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b733aee6-0587-4995-959d-8b63b408156c" containerName="kube-rbac-proxy"
Jul 07 12:03:16.881922 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.881905    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:16.887627 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.887602    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e7782-serving-cert\""
Jul 07 12:03:16.888217 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.888199    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e7782-kube-rbac-proxy-sar-config\""
Jul 07 12:03:16.895943 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.895924    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"]
Jul 07 12:03:16.931995 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.931974    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:03:16.983992 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.983963    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760e3751-35d0-46ba-9c8c-220f4daa29bb-openshift-service-ca-bundle\") pod \"ensemble-graph-e7782-6c5f799595-97lbg\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") " pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:16.984110 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:16.984016    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls\") pod \"ensemble-graph-e7782-6c5f799595-97lbg\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") " pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:17.084981 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:17.084955    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls\") pod \"ensemble-graph-e7782-6c5f799595-97lbg\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") " pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:17.085086 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:17.085013    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760e3751-35d0-46ba-9c8c-220f4daa29bb-openshift-service-ca-bundle\") pod \"ensemble-graph-e7782-6c5f799595-97lbg\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") " pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:17.085129 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:03:17.085109    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-e7782-serving-cert: secret "ensemble-graph-e7782-serving-cert" not found
Jul 07 12:03:17.085184 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:03:17.085175    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls podName:760e3751-35d0-46ba-9c8c-220f4daa29bb nodeName:}" failed. No retries permitted until 2026-07-07 12:03:17.585158862 +0000 UTC m=+1383.780735866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls") pod "ensemble-graph-e7782-6c5f799595-97lbg" (UID: "760e3751-35d0-46ba-9c8c-220f4daa29bb") : secret "ensemble-graph-e7782-serving-cert" not found
Jul 07 12:03:17.085579 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:17.085564    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760e3751-35d0-46ba-9c8c-220f4daa29bb-openshift-service-ca-bundle\") pod \"ensemble-graph-e7782-6c5f799595-97lbg\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") " pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:17.235674 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:17.235576    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:03:17.589466 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:17.589434    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls\") pod \"ensemble-graph-e7782-6c5f799595-97lbg\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") " pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:17.591801 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:17.591775    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls\") pod \"ensemble-graph-e7782-6c5f799595-97lbg\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") " pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:17.791355 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:17.791324    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:17.915788 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:17.915739    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"]
Jul 07 12:03:17.917958 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:03:17.917920    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760e3751_35d0_46ba_9c8c_220f4daa29bb.slice/crio-90808c54da4bd805c9458338d32d4424ebbc77f6851cf13c1a385f965fc2fa64 WatchSource:0}: Error finding container 90808c54da4bd805c9458338d32d4424ebbc77f6851cf13c1a385f965fc2fa64: Status 404 returned error can't find the container with id 90808c54da4bd805c9458338d32d4424ebbc77f6851cf13c1a385f965fc2fa64
Jul 07 12:03:18.780083 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:18.780051    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:03:18.780240 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:18.780141    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:03:18.859427 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:18.859393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" event={"ID":"760e3751-35d0-46ba-9c8c-220f4daa29bb","Type":"ContainerStarted","Data":"891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2"}
Jul 07 12:03:18.859587 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:18.859439    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" event={"ID":"760e3751-35d0-46ba-9c8c-220f4daa29bb","Type":"ContainerStarted","Data":"90808c54da4bd805c9458338d32d4424ebbc77f6851cf13c1a385f965fc2fa64"}
Jul 07 12:03:18.859587 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:18.859467    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:18.876242 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:18.876199    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" podStartSLOduration=2.876184479 podStartE2EDuration="2.876184479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:03:18.875373101 +0000 UTC m=+1385.070950123" watchObservedRunningTime="2026-07-07 12:03:18.876184479 +0000 UTC m=+1385.071761501"
Jul 07 12:03:23.780037 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:23.779997    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:03:24.828228 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:24.828194    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 12:03:24.869504 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:24.869475    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:03:28.780098 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:28.780059    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:03:33.779856 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:33.779821    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:03:34.828381 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:34.828348    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 12:03:36.519011 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.518991    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:03:36.628468 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.628441    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c59f328-1ff7-4e5a-8d2a-20f644472158-openshift-service-ca-bundle\") pod \"6c59f328-1ff7-4e5a-8d2a-20f644472158\" (UID: \"6c59f328-1ff7-4e5a-8d2a-20f644472158\") "
Jul 07 12:03:36.628618 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.628501    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c59f328-1ff7-4e5a-8d2a-20f644472158-proxy-tls\") pod \"6c59f328-1ff7-4e5a-8d2a-20f644472158\" (UID: \"6c59f328-1ff7-4e5a-8d2a-20f644472158\") "
Jul 07 12:03:36.628849 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.628824    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c59f328-1ff7-4e5a-8d2a-20f644472158-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6c59f328-1ff7-4e5a-8d2a-20f644472158" (UID: "6c59f328-1ff7-4e5a-8d2a-20f644472158"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:03:36.630439 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.630412    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c59f328-1ff7-4e5a-8d2a-20f644472158-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c59f328-1ff7-4e5a-8d2a-20f644472158" (UID: "6c59f328-1ff7-4e5a-8d2a-20f644472158"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:03:36.729201 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.729173    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c59f328-1ff7-4e5a-8d2a-20f644472158-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:03:36.729201 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.729197    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c59f328-1ff7-4e5a-8d2a-20f644472158-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:03:36.931477 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.931408    2572 generic.go:356] "Generic (PLEG): container finished" podID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerID="11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53" exitCode=0
Jul 07 12:03:36.931477 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.931457    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"
Jul 07 12:03:36.931477 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.931465    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" event={"ID":"6c59f328-1ff7-4e5a-8d2a-20f644472158","Type":"ContainerDied","Data":"11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53"}
Jul 07 12:03:36.931721 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.931510    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq" event={"ID":"6c59f328-1ff7-4e5a-8d2a-20f644472158","Type":"ContainerDied","Data":"6b4eec1762c6907dd165d67031ceaf3c102dc1de1019ad4f4721184d88f8610d"}
Jul 07 12:03:36.931721 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.931533    2572 scope.go:117] "RemoveContainer" containerID="11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53"
Jul 07 12:03:36.939729 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.939694    2572 scope.go:117] "RemoveContainer" containerID="11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53"
Jul 07 12:03:36.940051 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:03:36.940029    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53\": container with ID starting with 11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53 not found: ID does not exist" containerID="11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53"
Jul 07 12:03:36.940128 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.940063    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53"} err="failed to get container status \"11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53\": rpc error: code = NotFound desc = could not find container \"11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53\": container with ID starting with 11bc369af95bd010ce9288056e6afc617f1e8b37a28de29c732b68152e6c5f53 not found: ID does not exist"
Jul 07 12:03:36.954285 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.954265    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"]
Jul 07 12:03:36.957635 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:36.957615    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq"]
Jul 07 12:03:38.439029 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:38.438993    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" path="/var/lib/kubelet/pods/6c59f328-1ff7-4e5a-8d2a-20f644472158/volumes"
Jul 07 12:03:44.828462 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:44.828422    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 12:03:54.828588 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:03:54.828557    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:04:06.607645 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.607578    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"]
Jul 07 12:04:06.608068 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.607927    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899"
Jul 07 12:04:06.608068 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.607939    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899"
Jul 07 12:04:06.608068 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.608005    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c59f328-1ff7-4e5a-8d2a-20f644472158" containerName="sequence-graph-a4899"
Jul 07 12:04:06.610877 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.610862    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:06.613305 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.613287    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-b491e-kube-rbac-proxy-sar-config\""
Jul 07 12:04:06.613450 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.613390    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:04:06.613607 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.613565    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-b491e-serving-cert\""
Jul 07 12:04:06.618840 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.618816    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"]
Jul 07 12:04:06.643261 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.643242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls\") pod \"sequence-graph-b491e-5fcb67d95d-bqhpq\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") " pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:06.643360 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.643278    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68de497d-1ecc-4b3a-98b3-750aa10021c8-openshift-service-ca-bundle\") pod \"sequence-graph-b491e-5fcb67d95d-bqhpq\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") " pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:06.744426 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.744404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls\") pod \"sequence-graph-b491e-5fcb67d95d-bqhpq\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") " pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:06.744537 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.744449    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68de497d-1ecc-4b3a-98b3-750aa10021c8-openshift-service-ca-bundle\") pod \"sequence-graph-b491e-5fcb67d95d-bqhpq\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") " pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:06.744603 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:04:06.744541    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-b491e-serving-cert: secret "sequence-graph-b491e-serving-cert" not found
Jul 07 12:04:06.744603 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:04:06.744592    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls podName:68de497d-1ecc-4b3a-98b3-750aa10021c8 nodeName:}" failed. No retries permitted until 2026-07-07 12:04:07.244577617 +0000 UTC m=+1433.440154617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls") pod "sequence-graph-b491e-5fcb67d95d-bqhpq" (UID: "68de497d-1ecc-4b3a-98b3-750aa10021c8") : secret "sequence-graph-b491e-serving-cert" not found
Jul 07 12:04:06.745087 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:06.745067    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68de497d-1ecc-4b3a-98b3-750aa10021c8-openshift-service-ca-bundle\") pod \"sequence-graph-b491e-5fcb67d95d-bqhpq\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") " pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:07.238343 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:07.238315    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:04:07.248109 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:07.248085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls\") pod \"sequence-graph-b491e-5fcb67d95d-bqhpq\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") " pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:07.250321 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:07.250300    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls\") pod \"sequence-graph-b491e-5fcb67d95d-bqhpq\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") " pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:07.521840 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:07.521819    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:07.643501 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:07.643477    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"]
Jul 07 12:04:07.646187 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:04:07.646162    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68de497d_1ecc_4b3a_98b3_750aa10021c8.slice/crio-d5b0e5666cf0ffe92eaddf07e5e3d71a2dcafcbac5ae19738f90f838c92b8709 WatchSource:0}: Error finding container d5b0e5666cf0ffe92eaddf07e5e3d71a2dcafcbac5ae19738f90f838c92b8709: Status 404 returned error can't find the container with id d5b0e5666cf0ffe92eaddf07e5e3d71a2dcafcbac5ae19738f90f838c92b8709
Jul 07 12:04:08.048947 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:08.048912    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" event={"ID":"68de497d-1ecc-4b3a-98b3-750aa10021c8","Type":"ContainerStarted","Data":"9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c"}
Jul 07 12:04:08.049127 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:08.048963    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" event={"ID":"68de497d-1ecc-4b3a-98b3-750aa10021c8","Type":"ContainerStarted","Data":"d5b0e5666cf0ffe92eaddf07e5e3d71a2dcafcbac5ae19738f90f838c92b8709"}
Jul 07 12:04:08.049127 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:08.049048    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:04:08.074541 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:08.074495    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" podStartSLOduration=2.074480923 podStartE2EDuration="2.074480923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:04:08.072733874 +0000 UTC m=+1434.268310890" watchObservedRunningTime="2026-07-07 12:04:08.074480923 +0000 UTC m=+1434.270057944"
Jul 07 12:04:14.057871 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:04:14.057839    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:05:14.312121 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:05:14.312094    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:05:14.315206 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:05:14.315186    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:07:27.301232 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:07:27.301199    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:08:18.137806 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:08:18.137776    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:10:14.337203 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:10:14.337177    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:10:14.341989 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:10:14.341972    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:11:31.520382 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.520353    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"]
Jul 07 12:11:31.520866 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.520545    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782" containerID="cri-o://891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2" gracePeriod=30
Jul 07 12:11:31.582728 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.582697    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:11:31.757352 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.757323    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:11:31.816654 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.816587    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"]
Jul 07 12:11:31.819952 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.819937    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.822967 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.822945    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ec777-predictor-serving-cert\""
Jul 07 12:11:31.823137 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.823122    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ec777-kube-rbac-proxy-sar-config\""
Jul 07 12:11:31.830743 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.830719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.830844 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.830755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9f5\" (UniqueName: \"kubernetes.io/projected/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-api-access-sd9f5\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.830844 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.830787    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.851241 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.851217    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"]
Jul 07 12:11:31.852515 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.852493    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"]
Jul 07 12:11:31.852719 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.852698    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container" containerID="cri-o://ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be" gracePeriod=30
Jul 07 12:11:31.852812 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.852724    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kube-rbac-proxy" containerID="cri-o://d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b" gracePeriod=30
Jul 07 12:11:31.870178 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.870153    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:11:31.931742 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.931716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.931842 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.931747    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9f5\" (UniqueName: \"kubernetes.io/projected/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-api-access-sd9f5\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.931842 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.931772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.932007 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:11:31.931892    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ec777-predictor-serving-cert: secret "success-200-isvc-ec777-predictor-serving-cert" not found
Jul 07 12:11:31.932007 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:11:31.931951    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls podName:4d1ee5e4-b6d1-4191-84b0-c92b3300b075 nodeName:}" failed. No retries permitted until 2026-07-07 12:11:32.431931688 +0000 UTC m=+1878.627508701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls") pod "success-200-isvc-ec777-predictor-5c69cb57dc-l448d" (UID: "4d1ee5e4-b6d1-4191-84b0-c92b3300b075") : secret "success-200-isvc-ec777-predictor-serving-cert" not found
Jul 07 12:11:31.932368 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.932351    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.940206 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.940183    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9f5\" (UniqueName: \"kubernetes.io/projected/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-api-access-sd9f5\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:31.984243 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:31.984219    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:11:32.436346 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:32.436317    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:32.439005 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:32.438972    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls\") pod \"success-200-isvc-ec777-predictor-5c69cb57dc-l448d\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") " pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:32.730195 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:32.730105    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:32.733158 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:32.733123    2572 generic.go:356] "Generic (PLEG): container finished" podID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerID="d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b" exitCode=2
Jul 07 12:11:32.733318 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:32.733178    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" event={"ID":"f6142e9d-7af0-433f-b5cb-2aee7ceadacb","Type":"ContainerDied","Data":"d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b"}
Jul 07 12:11:32.764555 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:32.764498    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:11:32.831391 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:32.831316    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:11:32.860809 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:32.860761    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"]
Jul 07 12:11:32.864294 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:11:32.864260    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1ee5e4_b6d1_4191_84b0_c92b3300b075.slice/crio-0152852713fd3c402f4697ca056ad538fc06ee69d8282ab5c3d31f9be9a56c74 WatchSource:0}: Error finding container 0152852713fd3c402f4697ca056ad538fc06ee69d8282ab5c3d31f9be9a56c74: Status 404 returned error can't find the container with id 0152852713fd3c402f4697ca056ad538fc06ee69d8282ab5c3d31f9be9a56c74
Jul 07 12:11:33.615168 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.615128    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 07 12:11:33.620522 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.620499    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 12:11:33.739306 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.739275    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" event={"ID":"4d1ee5e4-b6d1-4191-84b0-c92b3300b075","Type":"ContainerStarted","Data":"96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8"}
Jul 07 12:11:33.739708 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.739332    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:33.739708 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.739353    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" event={"ID":"4d1ee5e4-b6d1-4191-84b0-c92b3300b075","Type":"ContainerStarted","Data":"bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e"}
Jul 07 12:11:33.739708 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.739371    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" event={"ID":"4d1ee5e4-b6d1-4191-84b0-c92b3300b075","Type":"ContainerStarted","Data":"0152852713fd3c402f4697ca056ad538fc06ee69d8282ab5c3d31f9be9a56c74"}
Jul 07 12:11:33.739708 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.739493    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:33.740600 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.740579    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 12:11:33.763575 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:33.763533    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podStartSLOduration=2.763521961 podStartE2EDuration="2.763521961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:11:33.761414208 +0000 UTC m=+1879.956991231" watchObservedRunningTime="2026-07-07 12:11:33.763521961 +0000 UTC m=+1879.959098982"
Jul 07 12:11:34.589298 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.589271    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:11:34.651676 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.651632    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-proxy-tls\") pod \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") "
Jul 07 12:11:34.651786 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.651687    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-rbac-proxy-sar-config\") pod \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") "
Jul 07 12:11:34.651786 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.651727    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28tfm\" (UniqueName: \"kubernetes.io/projected/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-api-access-28tfm\") pod \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\" (UID: \"f6142e9d-7af0-433f-b5cb-2aee7ceadacb\") "
Jul 07 12:11:34.652063 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.652041    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f6142e9d-7af0-433f-b5cb-2aee7ceadacb" (UID: "f6142e9d-7af0-433f-b5cb-2aee7ceadacb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:11:34.653636 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.653615    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f6142e9d-7af0-433f-b5cb-2aee7ceadacb" (UID: "f6142e9d-7af0-433f-b5cb-2aee7ceadacb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:11:34.653739 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.653675    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-api-access-28tfm" (OuterVolumeSpecName: "kube-api-access-28tfm") pod "f6142e9d-7af0-433f-b5cb-2aee7ceadacb" (UID: "f6142e9d-7af0-433f-b5cb-2aee7ceadacb"). InnerVolumeSpecName "kube-api-access-28tfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:11:34.744082 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.744019    2572 generic.go:356] "Generic (PLEG): container finished" podID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerID="ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be" exitCode=0
Jul 07 12:11:34.744082 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.744062    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" event={"ID":"f6142e9d-7af0-433f-b5cb-2aee7ceadacb","Type":"ContainerDied","Data":"ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be"}
Jul 07 12:11:34.744467 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.744075    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"
Jul 07 12:11:34.744467 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.744131    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2" event={"ID":"f6142e9d-7af0-433f-b5cb-2aee7ceadacb","Type":"ContainerDied","Data":"9b453481a1296c995d07c32941b3a1e802e03a0120cb69f39c25aae7c5275507"}
Jul 07 12:11:34.744467 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.744172    2572 scope.go:117] "RemoveContainer" containerID="d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b"
Jul 07 12:11:34.744923 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.744800    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 12:11:34.754209 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.754186    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:11:34.754302 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.754214    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:11:34.754302 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.754228    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tfm\" (UniqueName: \"kubernetes.io/projected/f6142e9d-7af0-433f-b5cb-2aee7ceadacb-kube-api-access-28tfm\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:11:34.755042 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.755025    2572 scope.go:117] "RemoveContainer" containerID="ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be"
Jul 07 12:11:34.762004 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.761988    2572 scope.go:117] "RemoveContainer" containerID="d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b"
Jul 07 12:11:34.762224 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:11:34.762208    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b\": container with ID starting with d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b not found: ID does not exist" containerID="d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b"
Jul 07 12:11:34.762269 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.762231    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b"} err="failed to get container status \"d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b\": rpc error: code = NotFound desc = could not find container \"d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b\": container with ID starting with d304ba4b466614ef23a8183e4b839360068c4222248be2bd067ca068599b709b not found: ID does not exist"
Jul 07 12:11:34.762269 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.762245    2572 scope.go:117] "RemoveContainer" containerID="ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be"
Jul 07 12:11:34.762487 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:11:34.762471    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be\": container with ID starting with ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be not found: ID does not exist" containerID="ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be"
Jul 07 12:11:34.762539 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.762490    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be"} err="failed to get container status \"ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be\": rpc error: code = NotFound desc = could not find container \"ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be\": container with ID starting with ec83bf4117d07ab0f030ecbba36f33ea0a03c44877dd597d244b2c9a48efa7be not found: ID does not exist"
Jul 07 12:11:34.766919 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.766899    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"]
Jul 07 12:11:34.770623 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.770606    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2"]
Jul 07 12:11:34.867774 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:34.867748    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:11:36.439308 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:36.439233    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" path="/var/lib/kubelet/pods/f6142e9d-7af0-433f-b5cb-2aee7ceadacb/volumes"
Jul 07 12:11:39.749021 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:39.748995    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:11:39.749457 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:39.749430    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 12:11:39.867473 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:39.867443    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:11:44.867139 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:44.867105    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:11:44.867531 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:44.867217    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:11:49.749573 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:49.749536    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 12:11:49.867510 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:49.867472    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:11:54.867506 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:54.867460    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:11:59.749397 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:59.749358    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 12:11:59.867762 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:11:59.867728    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:01.670049 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.670027    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:12:01.756114 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.756090    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760e3751-35d0-46ba-9c8c-220f4daa29bb-openshift-service-ca-bundle\") pod \"760e3751-35d0-46ba-9c8c-220f4daa29bb\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") "
Jul 07 12:12:01.756227 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.756150    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls\") pod \"760e3751-35d0-46ba-9c8c-220f4daa29bb\" (UID: \"760e3751-35d0-46ba-9c8c-220f4daa29bb\") "
Jul 07 12:12:01.756403 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.756379    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760e3751-35d0-46ba-9c8c-220f4daa29bb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "760e3751-35d0-46ba-9c8c-220f4daa29bb" (UID: "760e3751-35d0-46ba-9c8c-220f4daa29bb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:12:01.758046 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.758025    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "760e3751-35d0-46ba-9c8c-220f4daa29bb" (UID: "760e3751-35d0-46ba-9c8c-220f4daa29bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:12:01.856830 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.856756    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760e3751-35d0-46ba-9c8c-220f4daa29bb-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:01.856830 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.856778    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760e3751-35d0-46ba-9c8c-220f4daa29bb-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:01.859145 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.859116    2572 generic.go:356] "Generic (PLEG): container finished" podID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerID="891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2" exitCode=0
Jul 07 12:12:01.859246 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.859164    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"
Jul 07 12:12:01.859246 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.859199    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" event={"ID":"760e3751-35d0-46ba-9c8c-220f4daa29bb","Type":"ContainerDied","Data":"891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2"}
Jul 07 12:12:01.859354 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.859250    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg" event={"ID":"760e3751-35d0-46ba-9c8c-220f4daa29bb","Type":"ContainerDied","Data":"90808c54da4bd805c9458338d32d4424ebbc77f6851cf13c1a385f965fc2fa64"}
Jul 07 12:12:01.859354 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.859273    2572 scope.go:117] "RemoveContainer" containerID="891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2"
Jul 07 12:12:01.868419 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.868403    2572 scope.go:117] "RemoveContainer" containerID="891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2"
Jul 07 12:12:01.868696 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:01.868650    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2\": container with ID starting with 891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2 not found: ID does not exist" containerID="891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2"
Jul 07 12:12:01.868750 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.868706    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2"} err="failed to get container status \"891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2\": rpc error: code = NotFound desc = could not find container \"891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2\": container with ID starting with 891e61f2bebb5b0463329518a53d28a40377538288882f3da233dfade0975cf2 not found: ID does not exist"
Jul 07 12:12:01.881333 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.881315    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"]
Jul 07 12:12:01.885363 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:01.885344    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg"]
Jul 07 12:12:02.439033 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:02.439006    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" path="/var/lib/kubelet/pods/760e3751-35d0-46ba-9c8c-220f4daa29bb/volumes"
Jul 07 12:12:09.750146 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:09.750110    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 12:12:19.750371 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:19.750345    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:12:21.177356 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.177327    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"]
Jul 07 12:12:21.177801 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.177524    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e" containerID="cri-o://9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c" gracePeriod=30
Jul 07 12:12:21.202466 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.202433    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:21.314149 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.314120    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"]
Jul 07 12:12:21.314353 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.314326    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container" containerID="cri-o://cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef" gracePeriod=30
Jul 07 12:12:21.314465 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.314381    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kube-rbac-proxy" containerID="cri-o://5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa" gracePeriod=30
Jul 07 12:12:21.332959 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.332936    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"]
Jul 07 12:12:21.333237 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333226    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container"
Jul 07 12:12:21.333282 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333239    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container"
Jul 07 12:12:21.333282 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333252    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782"
Jul 07 12:12:21.333282 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333258    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782"
Jul 07 12:12:21.333282 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333269    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kube-rbac-proxy"
Jul 07 12:12:21.333282 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333274    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kube-rbac-proxy"
Jul 07 12:12:21.333448 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333326    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="760e3751-35d0-46ba-9c8c-220f4daa29bb" containerName="ensemble-graph-e7782"
Jul 07 12:12:21.333448 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333332    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kube-rbac-proxy"
Jul 07 12:12:21.333448 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.333338    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6142e9d-7af0-433f-b5cb-2aee7ceadacb" containerName="kserve-container"
Jul 07 12:12:21.336182 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.336163    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.339040 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.339022    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-beb62-kube-rbac-proxy-sar-config\""
Jul 07 12:12:21.339129 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.339024    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-beb62-predictor-serving-cert\""
Jul 07 12:12:21.341265 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.341246    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:21.348126 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.348101    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"]
Jul 07 12:12:21.447290 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.447258    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:21.502289 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.502261    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.502422 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.502319    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.502422 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.502373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82nc2\" (UniqueName: \"kubernetes.io/projected/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-api-access-82nc2\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.530017 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.529992    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:21.603092 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.603067    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82nc2\" (UniqueName: \"kubernetes.io/projected/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-api-access-82nc2\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.603201 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.603111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.603201 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.603177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.603301 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:21.603282    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-beb62-predictor-serving-cert: secret "success-200-isvc-beb62-predictor-serving-cert" not found
Jul 07 12:12:21.603370 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:21.603358    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls podName:91d36dfe-1452-4cdb-8e4e-59e877c9a71e nodeName:}" failed. No retries permitted until 2026-07-07 12:12:22.103338939 +0000 UTC m=+1928.298915954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls") pod "success-200-isvc-beb62-predictor-74847944ff-snqr2" (UID: "91d36dfe-1452-4cdb-8e4e-59e877c9a71e") : secret "success-200-isvc-beb62-predictor-serving-cert" not found
Jul 07 12:12:21.603758 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.603742    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.615184 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.615164    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82nc2\" (UniqueName: \"kubernetes.io/projected/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-api-access-82nc2\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:21.774704 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.774654    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:21.869094 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.869069    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:21.939899 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.939878    2572 generic.go:356] "Generic (PLEG): container finished" podID="baf1012b-7087-43b9-b457-481b96490086" containerID="5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa" exitCode=2
Jul 07 12:12:21.939995 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:21.939953    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" event={"ID":"baf1012b-7087-43b9-b457-481b96490086","Type":"ContainerDied","Data":"5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa"}
Jul 07 12:12:22.105917 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.105836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:22.108253 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.108226    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls\") pod \"success-200-isvc-beb62-predictor-74847944ff-snqr2\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") " pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:22.246695 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.246650    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:22.369752 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.369699    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"]
Jul 07 12:12:22.372352 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:12:22.372319    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d36dfe_1452_4cdb_8e4e_59e877c9a71e.slice/crio-6752adc0cd703c5b2e38708447332a59719d41c80baada664f94ebf5e5c2ceec WatchSource:0}: Error finding container 6752adc0cd703c5b2e38708447332a59719d41c80baada664f94ebf5e5c2ceec: Status 404 returned error can't find the container with id 6752adc0cd703c5b2e38708447332a59719d41c80baada664f94ebf5e5c2ceec
Jul 07 12:12:22.946201 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.946172    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" event={"ID":"91d36dfe-1452-4cdb-8e4e-59e877c9a71e","Type":"ContainerStarted","Data":"b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af"}
Jul 07 12:12:22.946381 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.946224    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" event={"ID":"91d36dfe-1452-4cdb-8e4e-59e877c9a71e","Type":"ContainerStarted","Data":"78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1"}
Jul 07 12:12:22.946381 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.946252    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" event={"ID":"91d36dfe-1452-4cdb-8e4e-59e877c9a71e","Type":"ContainerStarted","Data":"6752adc0cd703c5b2e38708447332a59719d41c80baada664f94ebf5e5c2ceec"}
Jul 07 12:12:22.946381 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.946324    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:22.971987 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:22.971925    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podStartSLOduration=1.971906115 podStartE2EDuration="1.971906115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:12:22.964940979 +0000 UTC m=+1929.160518015" watchObservedRunningTime="2026-07-07 12:12:22.971906115 +0000 UTC m=+1929.167483140"
Jul 07 12:12:23.950220 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:23.950195    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:23.951416 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:23.951384    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 12:12:24.057924 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.057887    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:24.155061 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.155038    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:12:24.322635 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.322610    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr9jk\" (UniqueName: \"kubernetes.io/projected/baf1012b-7087-43b9-b457-481b96490086-kube-api-access-gr9jk\") pod \"baf1012b-7087-43b9-b457-481b96490086\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") "
Jul 07 12:12:24.322788 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.322686    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls\") pod \"baf1012b-7087-43b9-b457-481b96490086\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") "
Jul 07 12:12:24.322788 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.322725    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf1012b-7087-43b9-b457-481b96490086-kube-rbac-proxy-sar-config\") pod \"baf1012b-7087-43b9-b457-481b96490086\" (UID: \"baf1012b-7087-43b9-b457-481b96490086\") "
Jul 07 12:12:24.323062 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.323040    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf1012b-7087-43b9-b457-481b96490086-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "baf1012b-7087-43b9-b457-481b96490086" (UID: "baf1012b-7087-43b9-b457-481b96490086"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:12:24.324496 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.324475    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf1012b-7087-43b9-b457-481b96490086-kube-api-access-gr9jk" (OuterVolumeSpecName: "kube-api-access-gr9jk") pod "baf1012b-7087-43b9-b457-481b96490086" (UID: "baf1012b-7087-43b9-b457-481b96490086"). InnerVolumeSpecName "kube-api-access-gr9jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:12:24.324587 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.324564    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "baf1012b-7087-43b9-b457-481b96490086" (UID: "baf1012b-7087-43b9-b457-481b96490086"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:12:24.423247 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.423226    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9jk\" (UniqueName: \"kubernetes.io/projected/baf1012b-7087-43b9-b457-481b96490086-kube-api-access-gr9jk\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:24.423247 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.423245    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf1012b-7087-43b9-b457-481b96490086-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:24.423370 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.423255    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf1012b-7087-43b9-b457-481b96490086-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:24.954784 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.954749    2572 generic.go:356] "Generic (PLEG): container finished" podID="baf1012b-7087-43b9-b457-481b96490086" containerID="cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef" exitCode=0
Jul 07 12:12:24.955173 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.954804    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"
Jul 07 12:12:24.955173 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.954834    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" event={"ID":"baf1012b-7087-43b9-b457-481b96490086","Type":"ContainerDied","Data":"cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef"}
Jul 07 12:12:24.955173 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.954893    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb" event={"ID":"baf1012b-7087-43b9-b457-481b96490086","Type":"ContainerDied","Data":"b328912df74fee85847beb78059702fe416b2bcdb2d582d048f04416e4cec43b"}
Jul 07 12:12:24.955173 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.954919    2572 scope.go:117] "RemoveContainer" containerID="5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa"
Jul 07 12:12:24.955481 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.955453    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 12:12:24.962777 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.962760    2572 scope.go:117] "RemoveContainer" containerID="cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef"
Jul 07 12:12:24.969695 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.969677    2572 scope.go:117] "RemoveContainer" containerID="5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa"
Jul 07 12:12:24.969916 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:24.969901    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa\": container with ID starting with 5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa not found: ID does not exist" containerID="5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa"
Jul 07 12:12:24.969962 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.969924    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa"} err="failed to get container status \"5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa\": rpc error: code = NotFound desc = could not find container \"5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa\": container with ID starting with 5e5d92250d8cd787592b8d979dc85128f72acc6d072db5ffc5e84cdeea7155aa not found: ID does not exist"
Jul 07 12:12:24.969962 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.969939    2572 scope.go:117] "RemoveContainer" containerID="cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef"
Jul 07 12:12:24.970164 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:24.970147    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef\": container with ID starting with cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef not found: ID does not exist" containerID="cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef"
Jul 07 12:12:24.970212 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.970168    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef"} err="failed to get container status \"cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef\": rpc error: code = NotFound desc = could not find container \"cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef\": container with ID starting with cf74aca648036a449029f0fe3bf87594f723e3cbe8204be973a98ceca08e4eef not found: ID does not exist"
Jul 07 12:12:24.975088 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.975069    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"]
Jul 07 12:12:24.980161 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:24.980142    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb"]
Jul 07 12:12:26.438081 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:26.438034    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf1012b-7087-43b9-b457-481b96490086" path="/var/lib/kubelet/pods/baf1012b-7087-43b9-b457-481b96490086/volumes"
Jul 07 12:12:29.057117 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:29.057084    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:29.960737 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:29.960711    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:12:29.961258 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:29.961225    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 12:12:31.736760 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.736723    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"]
Jul 07 12:12:31.737241 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.737026    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container"
Jul 07 12:12:31.737241 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.737035    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container"
Jul 07 12:12:31.737241 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.737052    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kube-rbac-proxy"
Jul 07 12:12:31.737241 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.737058    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kube-rbac-proxy"
Jul 07 12:12:31.737241 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.737109    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kserve-container"
Jul 07 12:12:31.737241 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.737119    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf1012b-7087-43b9-b457-481b96490086" containerName="kube-rbac-proxy"
Jul 07 12:12:31.740950 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.740931    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:31.743109 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.743092    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-ec777-serving-cert\""
Jul 07 12:12:31.743331 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.743313    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-ec777-kube-rbac-proxy-sar-config\""
Jul 07 12:12:31.750429 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.750408    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"]
Jul 07 12:12:31.755155 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.755129    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:31.876045 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.876024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10dc505-479e-48be-92a8-d6332f5a66b8-openshift-service-ca-bundle\") pod \"splitter-graph-ec777-577d754886-kfjm6\" (UID: \"d10dc505-479e-48be-92a8-d6332f5a66b8\") " pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:31.876159 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.876072    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d10dc505-479e-48be-92a8-d6332f5a66b8-proxy-tls\") pod \"splitter-graph-ec777-577d754886-kfjm6\" (UID: \"d10dc505-479e-48be-92a8-d6332f5a66b8\") " pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:31.977144 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.977117    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d10dc505-479e-48be-92a8-d6332f5a66b8-proxy-tls\") pod \"splitter-graph-ec777-577d754886-kfjm6\" (UID: \"d10dc505-479e-48be-92a8-d6332f5a66b8\") " pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:31.977252 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.977187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10dc505-479e-48be-92a8-d6332f5a66b8-openshift-service-ca-bundle\") pod \"splitter-graph-ec777-577d754886-kfjm6\" (UID: \"d10dc505-479e-48be-92a8-d6332f5a66b8\") " pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:31.977818 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.977798    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10dc505-479e-48be-92a8-d6332f5a66b8-openshift-service-ca-bundle\") pod \"splitter-graph-ec777-577d754886-kfjm6\" (UID: \"d10dc505-479e-48be-92a8-d6332f5a66b8\") " pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:31.979484 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:31.979469    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d10dc505-479e-48be-92a8-d6332f5a66b8-proxy-tls\") pod \"splitter-graph-ec777-577d754886-kfjm6\" (UID: \"d10dc505-479e-48be-92a8-d6332f5a66b8\") " pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:32.051686 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:32.051666    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:32.177598 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:32.177527    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"]
Jul 07 12:12:32.180062 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:12:32.180038    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10dc505_479e_48be_92a8_d6332f5a66b8.slice/crio-de57e2079a05a57990e03bc6dfcd18f7047bffd8f814ff563c0bcde757918555 WatchSource:0}: Error finding container de57e2079a05a57990e03bc6dfcd18f7047bffd8f814ff563c0bcde757918555: Status 404 returned error can't find the container with id de57e2079a05a57990e03bc6dfcd18f7047bffd8f814ff563c0bcde757918555
Jul 07 12:12:32.989082 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:32.989044    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" event={"ID":"d10dc505-479e-48be-92a8-d6332f5a66b8","Type":"ContainerStarted","Data":"2c92fa0992b42e849d8ef4a2b419b5d99e46ff9c06d1f5944d73a7ea9f2881ab"}
Jul 07 12:12:32.989438 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:32.989091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" event={"ID":"d10dc505-479e-48be-92a8-d6332f5a66b8","Type":"ContainerStarted","Data":"de57e2079a05a57990e03bc6dfcd18f7047bffd8f814ff563c0bcde757918555"}
Jul 07 12:12:32.989438 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:32.989170    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:33.008383 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:33.008346    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" podStartSLOduration=2.008333811 podStartE2EDuration="2.008333811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:12:33.008104148 +0000 UTC m=+1939.203681171" watchObservedRunningTime="2026-07-07 12:12:33.008333811 +0000 UTC m=+1939.203910832"
Jul 07 12:12:33.488818 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:33.488789    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:34.057444 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:34.057404    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:34.057902 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:34.057506    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:12:38.999246 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:38.999216    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:39.057186 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:39.057146    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:39.961928 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:39.961891    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 12:12:41.813181 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:41.813105    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"]
Jul 07 12:12:41.813643 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:41.813323    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777" containerID="cri-o://2c92fa0992b42e849d8ef4a2b419b5d99e46ff9c06d1f5944d73a7ea9f2881ab" gracePeriod=30
Jul 07 12:12:41.843916 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:41.843884    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:41.994593 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:41.994536    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"]
Jul 07 12:12:41.999679 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:41.994987    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container" containerID="cri-o://bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e" gracePeriod=30
Jul 07 12:12:41.999679 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:41.995273    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kube-rbac-proxy" containerID="cri-o://96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8" gracePeriod=30
Jul 07 12:12:42.028045 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.028023    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:42.047813 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.047788    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"]
Jul 07 12:12:42.052367 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.052347    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.054909 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.054891    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5d4cc-predictor-serving-cert\""
Jul 07 12:12:42.055329 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.055313    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5d4cc-kube-rbac-proxy-sar-config\""
Jul 07 12:12:42.064374 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.064323    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"]
Jul 07 12:12:42.140635 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.140606    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:42.147643 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.147621    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.147778 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.147758    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.147843 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.147812    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjts9\" (UniqueName: \"kubernetes.io/projected/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-api-access-jjts9\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.234452 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.234419    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:42.248518 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.248491    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.248647 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.248526    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjts9\" (UniqueName: \"kubernetes.io/projected/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-api-access-jjts9\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.248647 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.248547    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.248779 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:42.248671    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-serving-cert: secret "success-200-isvc-5d4cc-predictor-serving-cert" not found
Jul 07 12:12:42.248824 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:42.248792    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls podName:cfece5f1-c1bf-4d29-a081-a887c3a475ba nodeName:}" failed. No retries permitted until 2026-07-07 12:12:42.748771861 +0000 UTC m=+1948.944348878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls") pod "success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" (UID: "cfece5f1-c1bf-4d29-a081-a887c3a475ba") : secret "success-200-isvc-5d4cc-predictor-serving-cert" not found
Jul 07 12:12:42.249262 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.249236    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.271713 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.271650    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjts9\" (UniqueName: \"kubernetes.io/projected/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-api-access-jjts9\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.314423 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.314346    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:42.561142 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.561112    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:12:42.752235 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.752210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.754417 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.754396    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls\") pod \"success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") " pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:42.961870 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:42.961839    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:43.030942 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:43.030908    2572 generic.go:356] "Generic (PLEG): container finished" podID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerID="96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8" exitCode=2
Jul 07 12:12:43.031097 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:43.030999    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" event={"ID":"4d1ee5e4-b6d1-4191-84b0-c92b3300b075","Type":"ContainerDied","Data":"96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8"}
Jul 07 12:12:43.092714 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:43.092690    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"]
Jul 07 12:12:43.094959 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:12:43.094927    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfece5f1_c1bf_4d29_a081_a887c3a475ba.slice/crio-d2fe793cae3947471b34005e7bffcd3643466eb4d99ebb5dd7cdc9b2c3bbb3d1 WatchSource:0}: Error finding container d2fe793cae3947471b34005e7bffcd3643466eb4d99ebb5dd7cdc9b2c3bbb3d1: Status 404 returned error can't find the container with id d2fe793cae3947471b34005e7bffcd3643466eb4d99ebb5dd7cdc9b2c3bbb3d1
Jul 07 12:12:43.997779 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:43.997745    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:44.038809 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.038773    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" event={"ID":"cfece5f1-c1bf-4d29-a081-a887c3a475ba","Type":"ContainerStarted","Data":"7a17b55826aa550caae1199f4103e1dd4d986035c7aa4044b2c2949abb0a5c7a"}
Jul 07 12:12:44.038967 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.038846    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" event={"ID":"cfece5f1-c1bf-4d29-a081-a887c3a475ba","Type":"ContainerStarted","Data":"5464227c1e4b994b284a879a56769353c9c52b2acc12de8450858b20c99040f8"}
Jul 07 12:12:44.038967 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.038887    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:44.038967 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.038917    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" event={"ID":"cfece5f1-c1bf-4d29-a081-a887c3a475ba","Type":"ContainerStarted","Data":"d2fe793cae3947471b34005e7bffcd3643466eb4d99ebb5dd7cdc9b2c3bbb3d1"}
Jul 07 12:12:44.056565 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.056536    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:44.059798 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.059760    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podStartSLOduration=2.059747464 podStartE2EDuration="2.059747464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:12:44.057949474 +0000 UTC m=+1950.253526507" watchObservedRunningTime="2026-07-07 12:12:44.059747464 +0000 UTC m=+1950.255324485"
Jul 07 12:12:44.744952 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.744925    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 07 12:12:44.852206 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.852186    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:12:44.870802 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.870781    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-rbac-proxy-sar-config\") pod \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") "
Jul 07 12:12:44.870924 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.870848    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd9f5\" (UniqueName: \"kubernetes.io/projected/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-api-access-sd9f5\") pod \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") "
Jul 07 12:12:44.870924 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.870882    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls\") pod \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\" (UID: \"4d1ee5e4-b6d1-4191-84b0-c92b3300b075\") "
Jul 07 12:12:44.871435 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.871187    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d1ee5e4-b6d1-4191-84b0-c92b3300b075" (UID: "4d1ee5e4-b6d1-4191-84b0-c92b3300b075"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:12:44.872997 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.872965    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-api-access-sd9f5" (OuterVolumeSpecName: "kube-api-access-sd9f5") pod "4d1ee5e4-b6d1-4191-84b0-c92b3300b075" (UID: "4d1ee5e4-b6d1-4191-84b0-c92b3300b075"). InnerVolumeSpecName "kube-api-access-sd9f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:12:44.873102 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.873061    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d1ee5e4-b6d1-4191-84b0-c92b3300b075" (UID: "4d1ee5e4-b6d1-4191-84b0-c92b3300b075"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:12:44.971729 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.971670    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9f5\" (UniqueName: \"kubernetes.io/projected/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-api-access-sd9f5\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:44.971729 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.971697    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:44.971729 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:44.971710    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d1ee5e4-b6d1-4191-84b0-c92b3300b075-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:45.044206 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.044180    2572 generic.go:356] "Generic (PLEG): container finished" podID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerID="bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e" exitCode=0
Jul 07 12:12:45.044625 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.044238    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"
Jul 07 12:12:45.044625 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.044261    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" event={"ID":"4d1ee5e4-b6d1-4191-84b0-c92b3300b075","Type":"ContainerDied","Data":"bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e"}
Jul 07 12:12:45.044625 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.044323    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d" event={"ID":"4d1ee5e4-b6d1-4191-84b0-c92b3300b075","Type":"ContainerDied","Data":"0152852713fd3c402f4697ca056ad538fc06ee69d8282ab5c3d31f9be9a56c74"}
Jul 07 12:12:45.044625 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.044352    2572 scope.go:117] "RemoveContainer" containerID="96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8"
Jul 07 12:12:45.044891 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.044747    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:45.045869 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.045842    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 12:12:45.052561 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.052544    2572 scope.go:117] "RemoveContainer" containerID="bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e"
Jul 07 12:12:45.059629 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.059606    2572 scope.go:117] "RemoveContainer" containerID="96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8"
Jul 07 12:12:45.059912 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:45.059891    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8\": container with ID starting with 96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8 not found: ID does not exist" containerID="96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8"
Jul 07 12:12:45.059966 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.059921    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8"} err="failed to get container status \"96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8\": rpc error: code = NotFound desc = could not find container \"96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8\": container with ID starting with 96a69e31c00cbb0bc05ca2dd46ab1d06ab93c535e52387f7921fccec2c6b07b8 not found: ID does not exist"
Jul 07 12:12:45.059966 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.059937    2572 scope.go:117] "RemoveContainer" containerID="bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e"
Jul 07 12:12:45.060185 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:45.060168    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e\": container with ID starting with bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e not found: ID does not exist" containerID="bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e"
Jul 07 12:12:45.060230 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.060190    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e"} err="failed to get container status \"bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e\": rpc error: code = NotFound desc = could not find container \"bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e\": container with ID starting with bd5641510430d70a8a6afaacecb62bbaa13eec3a54435c5f84f3c7ecc82ac64e not found: ID does not exist"
Jul 07 12:12:45.065871 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.065851    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"]
Jul 07 12:12:45.070214 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:45.070197    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d"]
Jul 07 12:12:46.054901 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:46.054861    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 12:12:46.439214 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:46.439136    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" path="/var/lib/kubelet/pods/4d1ee5e4-b6d1-4191-84b0-c92b3300b075/volumes"
Jul 07 12:12:48.998401 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:48.998359    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:49.057237 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:49.057211    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:49.961777 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:49.961738    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 12:12:51.059759 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.059729    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:12:51.060150 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.060086    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 12:12:51.324828 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.324806    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:12:51.414443 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.414414    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68de497d-1ecc-4b3a-98b3-750aa10021c8-openshift-service-ca-bundle\") pod \"68de497d-1ecc-4b3a-98b3-750aa10021c8\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") "
Jul 07 12:12:51.414607 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.414450    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls\") pod \"68de497d-1ecc-4b3a-98b3-750aa10021c8\" (UID: \"68de497d-1ecc-4b3a-98b3-750aa10021c8\") "
Jul 07 12:12:51.414794 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.414770    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68de497d-1ecc-4b3a-98b3-750aa10021c8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "68de497d-1ecc-4b3a-98b3-750aa10021c8" (UID: "68de497d-1ecc-4b3a-98b3-750aa10021c8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:12:51.416424 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.416402    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68de497d-1ecc-4b3a-98b3-750aa10021c8" (UID: "68de497d-1ecc-4b3a-98b3-750aa10021c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:12:51.514974 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.514952    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68de497d-1ecc-4b3a-98b3-750aa10021c8-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:51.514974 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:51.514973    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68de497d-1ecc-4b3a-98b3-750aa10021c8-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:12:52.077439 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.077399    2572 generic.go:356] "Generic (PLEG): container finished" podID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerID="9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c" exitCode=0
Jul 07 12:12:52.077439 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.077450    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"
Jul 07 12:12:52.077987 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.077489    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" event={"ID":"68de497d-1ecc-4b3a-98b3-750aa10021c8","Type":"ContainerDied","Data":"9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c"}
Jul 07 12:12:52.077987 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.077555    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq" event={"ID":"68de497d-1ecc-4b3a-98b3-750aa10021c8","Type":"ContainerDied","Data":"d5b0e5666cf0ffe92eaddf07e5e3d71a2dcafcbac5ae19738f90f838c92b8709"}
Jul 07 12:12:52.077987 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.077590    2572 scope.go:117] "RemoveContainer" containerID="9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c"
Jul 07 12:12:52.089066 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.089042    2572 scope.go:117] "RemoveContainer" containerID="9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c"
Jul 07 12:12:52.089388 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:12:52.089367    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c\": container with ID starting with 9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c not found: ID does not exist" containerID="9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c"
Jul 07 12:12:52.089468 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.089395    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c"} err="failed to get container status \"9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c\": rpc error: code = NotFound desc = could not find container \"9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c\": container with ID starting with 9d61eb9b2e0c7609125d178eae29639e487528d45737f02fd745fc6e9679a72c not found: ID does not exist"
Jul 07 12:12:52.106473 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.106445    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"]
Jul 07 12:12:52.122344 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.122322    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq"]
Jul 07 12:12:52.438308 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:52.438215    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" path="/var/lib/kubelet/pods/68de497d-1ecc-4b3a-98b3-750aa10021c8/volumes"
Jul 07 12:12:53.997970 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:53.997935    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:53.998344 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:53.998053    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:12:58.998164 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:58.998119    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:12:59.961647 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:12:59.961603    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 12:13:01.060960 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:01.060918    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 12:13:03.997200 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:03.997163    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:13:08.997300 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:08.997264    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:13:09.962745 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:09.962717    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:13:11.060597 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:11.060563    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 12:13:12.163444 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.163369    2572 generic.go:356] "Generic (PLEG): container finished" podID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerID="2c92fa0992b42e849d8ef4a2b419b5d99e46ff9c06d1f5944d73a7ea9f2881ab" exitCode=0
Jul 07 12:13:12.163851 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.163439    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" event={"ID":"d10dc505-479e-48be-92a8-d6332f5a66b8","Type":"ContainerDied","Data":"2c92fa0992b42e849d8ef4a2b419b5d99e46ff9c06d1f5944d73a7ea9f2881ab"}
Jul 07 12:13:12.456647 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.456622    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:13:12.563462 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.563433    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10dc505-479e-48be-92a8-d6332f5a66b8-openshift-service-ca-bundle\") pod \"d10dc505-479e-48be-92a8-d6332f5a66b8\" (UID: \"d10dc505-479e-48be-92a8-d6332f5a66b8\") "
Jul 07 12:13:12.563616 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.563469    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d10dc505-479e-48be-92a8-d6332f5a66b8-proxy-tls\") pod \"d10dc505-479e-48be-92a8-d6332f5a66b8\" (UID: \"d10dc505-479e-48be-92a8-d6332f5a66b8\") "
Jul 07 12:13:12.563828 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.563805    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10dc505-479e-48be-92a8-d6332f5a66b8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d10dc505-479e-48be-92a8-d6332f5a66b8" (UID: "d10dc505-479e-48be-92a8-d6332f5a66b8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:13:12.565591 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.565565    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10dc505-479e-48be-92a8-d6332f5a66b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d10dc505-479e-48be-92a8-d6332f5a66b8" (UID: "d10dc505-479e-48be-92a8-d6332f5a66b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:13:12.664370 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.664345    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10dc505-479e-48be-92a8-d6332f5a66b8-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:13:12.664370 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:12.664365    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d10dc505-479e-48be-92a8-d6332f5a66b8-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:13:13.168313 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:13.168279    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"
Jul 07 12:13:13.168313 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:13.168304    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6" event={"ID":"d10dc505-479e-48be-92a8-d6332f5a66b8","Type":"ContainerDied","Data":"de57e2079a05a57990e03bc6dfcd18f7047bffd8f814ff563c0bcde757918555"}
Jul 07 12:13:13.168812 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:13.168361    2572 scope.go:117] "RemoveContainer" containerID="2c92fa0992b42e849d8ef4a2b419b5d99e46ff9c06d1f5944d73a7ea9f2881ab"
Jul 07 12:13:13.188600 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:13.188573    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"]
Jul 07 12:13:13.194975 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:13.194956    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6"]
Jul 07 12:13:14.439386 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:14.439354    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" path="/var/lib/kubelet/pods/d10dc505-479e-48be-92a8-d6332f5a66b8/volumes"
Jul 07 12:13:21.060697 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.060640    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 12:13:21.381593 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381517    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"]
Jul 07 12:13:21.381914 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381897    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777"
Jul 07 12:13:21.382003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381916    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777"
Jul 07 12:13:21.382003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381933    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kube-rbac-proxy"
Jul 07 12:13:21.382003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381941    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kube-rbac-proxy"
Jul 07 12:13:21.382003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381957    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container"
Jul 07 12:13:21.382003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381966    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container"
Jul 07 12:13:21.382003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381984    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e"
Jul 07 12:13:21.382003 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.381992    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e"
Jul 07 12:13:21.382376 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.382080    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10dc505-479e-48be-92a8-d6332f5a66b8" containerName="splitter-graph-ec777"
Jul 07 12:13:21.382376 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.382096    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kserve-container"
Jul 07 12:13:21.382376 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.382106    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1ee5e4-b6d1-4191-84b0-c92b3300b075" containerName="kube-rbac-proxy"
Jul 07 12:13:21.382376 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.382119    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="68de497d-1ecc-4b3a-98b3-750aa10021c8" containerName="sequence-graph-b491e"
Jul 07 12:13:21.384932 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.384913    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:21.387146 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.387126    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-beb62-serving-cert\""
Jul 07 12:13:21.387267 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.387230    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-beb62-kube-rbac-proxy-sar-config\""
Jul 07 12:13:21.388514 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.388490    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:13:21.395869 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.395848    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"]
Jul 07 12:13:21.427851 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.427828    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655adc5c-0a5b-43bf-a894-6743b04a0eba-openshift-service-ca-bundle\") pod \"switch-graph-beb62-5c7d9675f8-nbnkw\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") " pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:21.427937 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.427917    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls\") pod \"switch-graph-beb62-5c7d9675f8-nbnkw\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") " pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:21.528472 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.528446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls\") pod \"switch-graph-beb62-5c7d9675f8-nbnkw\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") " pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:21.528631 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.528498    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655adc5c-0a5b-43bf-a894-6743b04a0eba-openshift-service-ca-bundle\") pod \"switch-graph-beb62-5c7d9675f8-nbnkw\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") " pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:21.528769 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:13:21.528744    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-beb62-serving-cert: secret "switch-graph-beb62-serving-cert" not found
Jul 07 12:13:21.528971 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:13:21.528957    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls podName:655adc5c-0a5b-43bf-a894-6743b04a0eba nodeName:}" failed. No retries permitted until 2026-07-07 12:13:22.028939242 +0000 UTC m=+1988.224516246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls") pod "switch-graph-beb62-5c7d9675f8-nbnkw" (UID: "655adc5c-0a5b-43bf-a894-6743b04a0eba") : secret "switch-graph-beb62-serving-cert" not found
Jul 07 12:13:21.529134 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:21.529115    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655adc5c-0a5b-43bf-a894-6743b04a0eba-openshift-service-ca-bundle\") pod \"switch-graph-beb62-5c7d9675f8-nbnkw\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") " pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:22.031620 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:22.031590    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls\") pod \"switch-graph-beb62-5c7d9675f8-nbnkw\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") " pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:22.031797 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:13:22.031713    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-beb62-serving-cert: secret "switch-graph-beb62-serving-cert" not found
Jul 07 12:13:22.031797 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:13:22.031773    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls podName:655adc5c-0a5b-43bf-a894-6743b04a0eba nodeName:}" failed. No retries permitted until 2026-07-07 12:13:23.031759715 +0000 UTC m=+1989.227336716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls") pod "switch-graph-beb62-5c7d9675f8-nbnkw" (UID: "655adc5c-0a5b-43bf-a894-6743b04a0eba") : secret "switch-graph-beb62-serving-cert" not found
Jul 07 12:13:22.096969 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:22.096938    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:13:23.037600 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:23.037571    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls\") pod \"switch-graph-beb62-5c7d9675f8-nbnkw\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") " pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:23.040043 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:23.040017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls\") pod \"switch-graph-beb62-5c7d9675f8-nbnkw\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") " pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:23.195030 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:23.194999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:23.323195 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:23.323173    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"]
Jul 07 12:13:23.325782 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:13:23.325757    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655adc5c_0a5b_43bf_a894_6743b04a0eba.slice/crio-d52a2bc1b9ba3be242da5b3168175d2f9294ec1ccd5faae18ba04cf49050da30 WatchSource:0}: Error finding container d52a2bc1b9ba3be242da5b3168175d2f9294ec1ccd5faae18ba04cf49050da30: Status 404 returned error can't find the container with id d52a2bc1b9ba3be242da5b3168175d2f9294ec1ccd5faae18ba04cf49050da30
Jul 07 12:13:24.211322 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:24.211288    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" event={"ID":"655adc5c-0a5b-43bf-a894-6743b04a0eba","Type":"ContainerStarted","Data":"1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb"}
Jul 07 12:13:24.211695 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:24.211336    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" event={"ID":"655adc5c-0a5b-43bf-a894-6743b04a0eba","Type":"ContainerStarted","Data":"d52a2bc1b9ba3be242da5b3168175d2f9294ec1ccd5faae18ba04cf49050da30"}
Jul 07 12:13:24.211695 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:24.211367    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:24.231750 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:24.231705    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" podStartSLOduration=3.231690579 podStartE2EDuration="3.231690579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:13:24.230334627 +0000 UTC m=+1990.425911650" watchObservedRunningTime="2026-07-07 12:13:24.231690579 +0000 UTC m=+1990.427267600"
Jul 07 12:13:30.220767 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:30.220740    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:13:31.061210 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:31.061180    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:13:32.059485 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:32.059445    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:13:38.677282 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:38.677248    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:13:42.022577 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.022549    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"]
Jul 07 12:13:42.025986 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.025964    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:42.027207 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.027182    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:13:42.028512 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.028479    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5d4cc-kube-rbac-proxy-sar-config\""
Jul 07 12:13:42.028637 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.028621    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5d4cc-serving-cert\""
Jul 07 12:13:42.037170 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.037147    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"]
Jul 07 12:13:42.127013 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.126986    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:13:42.176615 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.176592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-openshift-service-ca-bundle\") pod \"splitter-graph-5d4cc-9fb6994f-tzflz\" (UID: \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\") " pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:42.176730 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.176628    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-proxy-tls\") pod \"splitter-graph-5d4cc-9fb6994f-tzflz\" (UID: \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\") " pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:42.277977 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.277917    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-openshift-service-ca-bundle\") pod \"splitter-graph-5d4cc-9fb6994f-tzflz\" (UID: \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\") " pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:42.277977 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.277970    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-proxy-tls\") pod \"splitter-graph-5d4cc-9fb6994f-tzflz\" (UID: \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\") " pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:42.278512 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.278494    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-openshift-service-ca-bundle\") pod \"splitter-graph-5d4cc-9fb6994f-tzflz\" (UID: \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\") " pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:42.280354 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.280335    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-proxy-tls\") pod \"splitter-graph-5d4cc-9fb6994f-tzflz\" (UID: \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\") " pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:42.336127 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.336110    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:42.465672 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:42.465636    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"]
Jul 07 12:13:42.467761 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:13:42.467734    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0e2c70_fbd2_4fc3_b745_2df3813d483e.slice/crio-62197faaf44a404f3a63986cd725af7da1ec703af7a252f5901ac08d64eb6dbe WatchSource:0}: Error finding container 62197faaf44a404f3a63986cd725af7da1ec703af7a252f5901ac08d64eb6dbe: Status 404 returned error can't find the container with id 62197faaf44a404f3a63986cd725af7da1ec703af7a252f5901ac08d64eb6dbe
Jul 07 12:13:43.289688 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:43.289643    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" event={"ID":"cc0e2c70-fbd2-4fc3-b745-2df3813d483e","Type":"ContainerStarted","Data":"6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5"}
Jul 07 12:13:43.290075 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:43.289707    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" event={"ID":"cc0e2c70-fbd2-4fc3-b745-2df3813d483e","Type":"ContainerStarted","Data":"62197faaf44a404f3a63986cd725af7da1ec703af7a252f5901ac08d64eb6dbe"}
Jul 07 12:13:43.290075 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:43.289772    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:13:43.307970 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:43.307922    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" podStartSLOduration=1.307905681 podStartE2EDuration="1.307905681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:13:43.306502676 +0000 UTC m=+2009.502079699" watchObservedRunningTime="2026-07-07 12:13:43.307905681 +0000 UTC m=+2009.503482702"
Jul 07 12:13:49.300321 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:13:49.300286    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:14:32.093528 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:14:32.093490    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:14:32.193984 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:14:32.193950    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:15:01.440111 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:15:01.440079    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:15:14.362119 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:15:14.362077    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:15:14.367092 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:15:14.367075    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:19:33.042341 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:19:33.042304    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:20:14.386238 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:20:14.386213    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:20:14.391639 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:20:14.391623    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:21:56.701833 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:56.701793    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"]
Jul 07 12:21:56.702394 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:56.702044    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc" containerID="cri-o://6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5" gracePeriod=30
Jul 07 12:21:56.762415 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:56.762389    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:21:56.830748 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:56.830719    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"]
Jul 07 12:21:56.831095 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:56.831042    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kube-rbac-proxy" containerID="cri-o://7a17b55826aa550caae1199f4103e1dd4d986035c7aa4044b2c2949abb0a5c7a" gracePeriod=30
Jul 07 12:21:56.831234 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:56.831142    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container" containerID="cri-o://5464227c1e4b994b284a879a56769353c9c52b2acc12de8450858b20c99040f8" gracePeriod=30
Jul 07 12:21:56.861469 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:56.861440    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:21:56.919276 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:56.919249    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:21:57.155796 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:57.155771    2572 generic.go:356] "Generic (PLEG): container finished" podID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerID="7a17b55826aa550caae1199f4103e1dd4d986035c7aa4044b2c2949abb0a5c7a" exitCode=2
Jul 07 12:21:57.155951 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:57.155849    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" event={"ID":"cfece5f1-c1bf-4d29-a081-a887c3a475ba","Type":"ContainerDied","Data":"7a17b55826aa550caae1199f4103e1dd4d986035c7aa4044b2c2949abb0a5c7a"}
Jul 07 12:21:59.164404 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.164380    2572 generic.go:356] "Generic (PLEG): container finished" podID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerID="5464227c1e4b994b284a879a56769353c9c52b2acc12de8450858b20c99040f8" exitCode=0
Jul 07 12:21:59.164727 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.164448    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" event={"ID":"cfece5f1-c1bf-4d29-a081-a887c3a475ba","Type":"ContainerDied","Data":"5464227c1e4b994b284a879a56769353c9c52b2acc12de8450858b20c99040f8"}
Jul 07 12:21:59.271537 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.271517    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:21:59.298162 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.298080    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:21:59.372232 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.372207    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjts9\" (UniqueName: \"kubernetes.io/projected/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-api-access-jjts9\") pod \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") "
Jul 07 12:21:59.372332 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.372253    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls\") pod \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") "
Jul 07 12:21:59.372332 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.372300    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-rbac-proxy-sar-config\") pod \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\" (UID: \"cfece5f1-c1bf-4d29-a081-a887c3a475ba\") "
Jul 07 12:21:59.372707 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.372654    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cfece5f1-c1bf-4d29-a081-a887c3a475ba" (UID: "cfece5f1-c1bf-4d29-a081-a887c3a475ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:21:59.374015 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.373996    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-api-access-jjts9" (OuterVolumeSpecName: "kube-api-access-jjts9") pod "cfece5f1-c1bf-4d29-a081-a887c3a475ba" (UID: "cfece5f1-c1bf-4d29-a081-a887c3a475ba"). InnerVolumeSpecName "kube-api-access-jjts9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:21:59.374106 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.374062    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cfece5f1-c1bf-4d29-a081-a887c3a475ba" (UID: "cfece5f1-c1bf-4d29-a081-a887c3a475ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:21:59.473173 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.473155    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfece5f1-c1bf-4d29-a081-a887c3a475ba-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:21:59.473173 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.473174    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:21:59.473300 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:21:59.473185    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjts9\" (UniqueName: \"kubernetes.io/projected/cfece5f1-c1bf-4d29-a081-a887c3a475ba-kube-api-access-jjts9\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:22:00.172822 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:00.172788    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh" event={"ID":"cfece5f1-c1bf-4d29-a081-a887c3a475ba","Type":"ContainerDied","Data":"d2fe793cae3947471b34005e7bffcd3643466eb4d99ebb5dd7cdc9b2c3bbb3d1"}
Jul 07 12:22:00.173226 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:00.172854    2572 scope.go:117] "RemoveContainer" containerID="7a17b55826aa550caae1199f4103e1dd4d986035c7aa4044b2c2949abb0a5c7a"
Jul 07 12:22:00.173226 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:00.172798    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"
Jul 07 12:22:00.182131 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:00.182108    2572 scope.go:117] "RemoveContainer" containerID="5464227c1e4b994b284a879a56769353c9c52b2acc12de8450858b20c99040f8"
Jul 07 12:22:00.194921 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:00.194897    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"]
Jul 07 12:22:00.199416 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:00.199394    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh"]
Jul 07 12:22:00.438474 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:00.438412    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" path="/var/lib/kubelet/pods/cfece5f1-c1bf-4d29-a081-a887c3a475ba/volumes"
Jul 07 12:22:04.297978 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:04.297943    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:22:09.297833 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:09.297795    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:22:09.298225 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:09.297910    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:22:14.298428 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:14.298394    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:22:19.298114 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:19.298064    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:22:24.298056 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:24.298011    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:22:26.841057 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:26.841035    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:22:26.962694 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:26.962610    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-proxy-tls\") pod \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\" (UID: \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\") "
Jul 07 12:22:26.962694 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:26.962651    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-openshift-service-ca-bundle\") pod \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\" (UID: \"cc0e2c70-fbd2-4fc3-b745-2df3813d483e\") "
Jul 07 12:22:26.963047 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:26.963024    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cc0e2c70-fbd2-4fc3-b745-2df3813d483e" (UID: "cc0e2c70-fbd2-4fc3-b745-2df3813d483e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:22:26.964634 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:26.964616    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc0e2c70-fbd2-4fc3-b745-2df3813d483e" (UID: "cc0e2c70-fbd2-4fc3-b745-2df3813d483e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:22:27.063745 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.063720    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:22:27.063745 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.063742    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0e2c70-fbd2-4fc3-b745-2df3813d483e-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:22:27.274832 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.274806    2572 generic.go:356] "Generic (PLEG): container finished" podID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerID="6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5" exitCode=0
Jul 07 12:22:27.274948 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.274846    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"
Jul 07 12:22:27.274948 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.274860    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" event={"ID":"cc0e2c70-fbd2-4fc3-b745-2df3813d483e","Type":"ContainerDied","Data":"6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5"}
Jul 07 12:22:27.274948 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.274900    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz" event={"ID":"cc0e2c70-fbd2-4fc3-b745-2df3813d483e","Type":"ContainerDied","Data":"62197faaf44a404f3a63986cd725af7da1ec703af7a252f5901ac08d64eb6dbe"}
Jul 07 12:22:27.274948 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.274920    2572 scope.go:117] "RemoveContainer" containerID="6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5"
Jul 07 12:22:27.283050 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.283035    2572 scope.go:117] "RemoveContainer" containerID="6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5"
Jul 07 12:22:27.283293 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:22:27.283277    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5\": container with ID starting with 6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5 not found: ID does not exist" containerID="6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5"
Jul 07 12:22:27.283336 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.283301    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5"} err="failed to get container status \"6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5\": rpc error: code = NotFound desc = could not find container \"6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5\": container with ID starting with 6d139432c342c00200f38de500457fd7ff2a4f49c265632dfe0fef25778ab7b5 not found: ID does not exist"
Jul 07 12:22:27.294497 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.294473    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"]
Jul 07 12:22:27.298276 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:27.298255    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz"]
Jul 07 12:22:28.438242 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:28.438211    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" path="/var/lib/kubelet/pods/cc0e2c70-fbd2-4fc3-b745-2df3813d483e/volumes"
Jul 07 12:22:56.632136 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:22:56.632104    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:25:14.418196 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:25:14.418166    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:25:14.422464 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:25:14.422446    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:25:27.365561 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:25:27.365527    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:29:40.504800 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:40.504723    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"]
Jul 07 12:29:40.505345 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:40.504960    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62" containerID="cri-o://1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb" gracePeriod=30
Jul 07 12:29:40.627203 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:40.627174    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:29:40.829396 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:40.829324    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"]
Jul 07 12:29:40.829568 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:40.829545    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container" containerID="cri-o://78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1" gracePeriod=30
Jul 07 12:29:40.829648 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:40.829579    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kube-rbac-proxy" containerID="cri-o://b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af" gracePeriod=30
Jul 07 12:29:40.842871 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:40.842854    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:29:40.919986 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:40.919958    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:29:41.604562 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604534    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hmqqr/must-gather-wh6nx"]
Jul 07 12:29:41.604933 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604844    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container"
Jul 07 12:29:41.604933 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604856    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container"
Jul 07 12:29:41.604933 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604863    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc"
Jul 07 12:29:41.604933 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604869    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc"
Jul 07 12:29:41.604933 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604889    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kube-rbac-proxy"
Jul 07 12:29:41.604933 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604895    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kube-rbac-proxy"
Jul 07 12:29:41.604933 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604936    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kserve-container"
Jul 07 12:29:41.605151 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604947    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0e2c70-fbd2-4fc3-b745-2df3813d483e" containerName="splitter-graph-5d4cc"
Jul 07 12:29:41.605151 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.604953    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfece5f1-c1bf-4d29-a081-a887c3a475ba" containerName="kube-rbac-proxy"
Jul 07 12:29:41.609004 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.608976    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:29:41.611728 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.611705    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hmqqr\"/\"kube-root-ca.crt\""
Jul 07 12:29:41.611879 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.611757    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hmqqr\"/\"default-dockercfg-ddqzc\""
Jul 07 12:29:41.612334 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.612316    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hmqqr\"/\"openshift-service-ca.crt\""
Jul 07 12:29:41.615783 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.615762    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hmqqr/must-gather-wh6nx"]
Jul 07 12:29:41.621229 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.621178    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:29:41.680923 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.680901    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:29:41.685887 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.685867    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlnt8\" (UniqueName: \"kubernetes.io/projected/e5c64d46-806e-4a4f-813b-63d6d7f5643e-kube-api-access-vlnt8\") pod \"must-gather-wh6nx\" (UID: \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\") " pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:29:41.685955 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.685900    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5c64d46-806e-4a4f-813b-63d6d7f5643e-must-gather-output\") pod \"must-gather-wh6nx\" (UID: \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\") " pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:29:41.787126 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.787102    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlnt8\" (UniqueName: \"kubernetes.io/projected/e5c64d46-806e-4a4f-813b-63d6d7f5643e-kube-api-access-vlnt8\") pod \"must-gather-wh6nx\" (UID: \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\") " pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:29:41.787209 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.787137    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5c64d46-806e-4a4f-813b-63d6d7f5643e-must-gather-output\") pod \"must-gather-wh6nx\" (UID: \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\") " pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:29:41.787470 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.787455    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5c64d46-806e-4a4f-813b-63d6d7f5643e-must-gather-output\") pod \"must-gather-wh6nx\" (UID: \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\") " pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:29:41.795615 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.795594    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlnt8\" (UniqueName: \"kubernetes.io/projected/e5c64d46-806e-4a4f-813b-63d6d7f5643e-kube-api-access-vlnt8\") pod \"must-gather-wh6nx\" (UID: \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\") " pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:29:41.878109 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.878043    2572 generic.go:356] "Generic (PLEG): container finished" podID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerID="b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af" exitCode=2
Jul 07 12:29:41.878109 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.878082    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" event={"ID":"91d36dfe-1452-4cdb-8e4e-59e877c9a71e","Type":"ContainerDied","Data":"b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af"}
Jul 07 12:29:41.928440 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:41.928408    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:29:42.047675 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:42.047633    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hmqqr/must-gather-wh6nx"]
Jul 07 12:29:42.050531 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:29:42.050502    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c64d46_806e_4a4f_813b_63d6d7f5643e.slice/crio-5beee0979ccd9b81fb7847e9a397c5254cc6b9ed85c7c8ab15bfee371b59c8ec WatchSource:0}: Error finding container 5beee0979ccd9b81fb7847e9a397c5254cc6b9ed85c7c8ab15bfee371b59c8ec: Status 404 returned error can't find the container with id 5beee0979ccd9b81fb7847e9a397c5254cc6b9ed85c7c8ab15bfee371b59c8ec
Jul 07 12:29:42.052273 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:42.052254    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 12:29:42.885047 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:42.884974    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hmqqr/must-gather-wh6nx" event={"ID":"e5c64d46-806e-4a4f-813b-63d6d7f5643e","Type":"ContainerStarted","Data":"5beee0979ccd9b81fb7847e9a397c5254cc6b9ed85c7c8ab15bfee371b59c8ec"}
Jul 07 12:29:43.823310 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.823284    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:29:43.891056 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.891005    2572 generic.go:356] "Generic (PLEG): container finished" podID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerID="78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1" exitCode=0
Jul 07 12:29:43.891421 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.891094    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" event={"ID":"91d36dfe-1452-4cdb-8e4e-59e877c9a71e","Type":"ContainerDied","Data":"78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1"}
Jul 07 12:29:43.891421 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.891131    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"
Jul 07 12:29:43.891421 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.891157    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2" event={"ID":"91d36dfe-1452-4cdb-8e4e-59e877c9a71e","Type":"ContainerDied","Data":"6752adc0cd703c5b2e38708447332a59719d41c80baada664f94ebf5e5c2ceec"}
Jul 07 12:29:43.891421 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.891183    2572 scope.go:117] "RemoveContainer" containerID="b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af"
Jul 07 12:29:43.901967 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.901921    2572 scope.go:117] "RemoveContainer" containerID="78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1"
Jul 07 12:29:43.904280 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.904230    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82nc2\" (UniqueName: \"kubernetes.io/projected/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-api-access-82nc2\") pod \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") "
Jul 07 12:29:43.904404 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.904302    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-rbac-proxy-sar-config\") pod \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") "
Jul 07 12:29:43.904404 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.904351    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls\") pod \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\" (UID: \"91d36dfe-1452-4cdb-8e4e-59e877c9a71e\") "
Jul 07 12:29:43.904830 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.904802    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91d36dfe-1452-4cdb-8e4e-59e877c9a71e" (UID: "91d36dfe-1452-4cdb-8e4e-59e877c9a71e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:29:43.906977 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.906928    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-api-access-82nc2" (OuterVolumeSpecName: "kube-api-access-82nc2") pod "91d36dfe-1452-4cdb-8e4e-59e877c9a71e" (UID: "91d36dfe-1452-4cdb-8e4e-59e877c9a71e"). InnerVolumeSpecName "kube-api-access-82nc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:29:43.907156 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.907133    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91d36dfe-1452-4cdb-8e4e-59e877c9a71e" (UID: "91d36dfe-1452-4cdb-8e4e-59e877c9a71e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:29:43.911864 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.911776    2572 scope.go:117] "RemoveContainer" containerID="b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af"
Jul 07 12:29:43.912183 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:29:43.912055    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af\": container with ID starting with b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af not found: ID does not exist" containerID="b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af"
Jul 07 12:29:43.912183 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.912088    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af"} err="failed to get container status \"b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af\": rpc error: code = NotFound desc = could not find container \"b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af\": container with ID starting with b7da1757499759b741d5306504e3f1ed6a9e7316f748892c038219628bd0f4af not found: ID does not exist"
Jul 07 12:29:43.912183 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.912110    2572 scope.go:117] "RemoveContainer" containerID="78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1"
Jul 07 12:29:43.912535 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:29:43.912466    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1\": container with ID starting with 78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1 not found: ID does not exist" containerID="78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1"
Jul 07 12:29:43.912535 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:43.912495    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1"} err="failed to get container status \"78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1\": rpc error: code = NotFound desc = could not find container \"78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1\": container with ID starting with 78254c6b31778c4fb49aa69040c49a66f0ac840510b4c12ce2bd0806c9509de1 not found: ID does not exist"
Jul 07 12:29:44.005385 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:44.005348    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82nc2\" (UniqueName: \"kubernetes.io/projected/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-api-access-82nc2\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:29:44.005520 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:44.005380    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:29:44.005520 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:44.005463    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d36dfe-1452-4cdb-8e4e-59e877c9a71e-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:29:44.217493 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:44.217466    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"]
Jul 07 12:29:44.221426 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:44.221402    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2"]
Jul 07 12:29:44.439823 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:44.439794    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" path="/var/lib/kubelet/pods/91d36dfe-1452-4cdb-8e4e-59e877c9a71e/volumes"
Jul 07 12:29:45.221035 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:45.220993    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:29:48.913447 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:48.913408    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hmqqr/must-gather-wh6nx" event={"ID":"e5c64d46-806e-4a4f-813b-63d6d7f5643e","Type":"ContainerStarted","Data":"160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023"}
Jul 07 12:29:48.913866 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:48.913457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hmqqr/must-gather-wh6nx" event={"ID":"e5c64d46-806e-4a4f-813b-63d6d7f5643e","Type":"ContainerStarted","Data":"59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82"}
Jul 07 12:29:48.928935 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:48.928892    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hmqqr/must-gather-wh6nx" podStartSLOduration=1.900143006 podStartE2EDuration="7.928878514s" totalImagesPullingTime="6.028735508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:29:48.927956722 +0000 UTC m=+2975.123533745" watchObservedRunningTime="2026-07-07 12:29:48.928878514 +0000 UTC m=+2975.124455560"
Jul 07 12:29:50.220257 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:50.220219    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:29:55.220311 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:55.220255    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:29:55.220831 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:55.220380    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:29:56.639247 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:56.639217    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:29:57.516981 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:57.516944    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:29:58.347855 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:58.347820    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:29:59.153449 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:59.153416    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:29:59.942211 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:29:59.942165    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:00.219965 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:00.219888    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:30:00.712338 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:00.712311    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:01.478886 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:01.478855    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:02.236601 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:02.236569    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:03.025012 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:03.024974    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:03.798380 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:03.798348    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:04.600560 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:04.600528    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:05.220818 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:05.220784    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:30:05.395682 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:05.395630    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:06.182068 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:06.182030    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:06.982028 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:06.981998    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb62-5c7d9675f8-nbnkw_655adc5c-0a5b-43bf-a894-6743b04a0eba/switch-graph-beb62/0.log"
Jul 07 12:30:10.219312 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:10.219281    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 12:30:10.648670 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:10.648640    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:30:10.720867 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:10.720842    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655adc5c-0a5b-43bf-a894-6743b04a0eba-openshift-service-ca-bundle\") pod \"655adc5c-0a5b-43bf-a894-6743b04a0eba\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") "
Jul 07 12:30:10.720976 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:10.720905    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls\") pod \"655adc5c-0a5b-43bf-a894-6743b04a0eba\" (UID: \"655adc5c-0a5b-43bf-a894-6743b04a0eba\") "
Jul 07 12:30:10.721196 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:10.721174    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655adc5c-0a5b-43bf-a894-6743b04a0eba-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "655adc5c-0a5b-43bf-a894-6743b04a0eba" (UID: "655adc5c-0a5b-43bf-a894-6743b04a0eba"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:30:10.722842 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:10.722824    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "655adc5c-0a5b-43bf-a894-6743b04a0eba" (UID: "655adc5c-0a5b-43bf-a894-6743b04a0eba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:30:10.822368 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:10.822314    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655adc5c-0a5b-43bf-a894-6743b04a0eba-openshift-service-ca-bundle\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:30:10.822368 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:10.822335    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655adc5c-0a5b-43bf-a894-6743b04a0eba-proxy-tls\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:30:11.009563 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.009535    2572 generic.go:356] "Generic (PLEG): container finished" podID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerID="59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82" exitCode=0
Jul 07 12:30:11.009700 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.009609    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hmqqr/must-gather-wh6nx" event={"ID":"e5c64d46-806e-4a4f-813b-63d6d7f5643e","Type":"ContainerDied","Data":"59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82"}
Jul 07 12:30:11.009945 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.009924    2572 scope.go:117] "RemoveContainer" containerID="59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82"
Jul 07 12:30:11.010887 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.010869    2572 generic.go:356] "Generic (PLEG): container finished" podID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerID="1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb" exitCode=0
Jul 07 12:30:11.010963 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.010914    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" event={"ID":"655adc5c-0a5b-43bf-a894-6743b04a0eba","Type":"ContainerDied","Data":"1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb"}
Jul 07 12:30:11.011015 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.010973    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw" event={"ID":"655adc5c-0a5b-43bf-a894-6743b04a0eba","Type":"ContainerDied","Data":"d52a2bc1b9ba3be242da5b3168175d2f9294ec1ccd5faae18ba04cf49050da30"}
Jul 07 12:30:11.011015 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.010916    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"
Jul 07 12:30:11.011090 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.010993    2572 scope.go:117] "RemoveContainer" containerID="1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb"
Jul 07 12:30:11.019868 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.019851    2572 scope.go:117] "RemoveContainer" containerID="1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb"
Jul 07 12:30:11.020112 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:30:11.020095    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb\": container with ID starting with 1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb not found: ID does not exist" containerID="1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb"
Jul 07 12:30:11.020172 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.020117    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb"} err="failed to get container status \"1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb\": rpc error: code = NotFound desc = could not find container \"1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb\": container with ID starting with 1791a30a426810f8458810966216fc10c7d262a3ede88c5c4f4e579957609bcb not found: ID does not exist"
Jul 07 12:30:11.037928 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.037906    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"]
Jul 07 12:30:11.042480 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.042459    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw"]
Jul 07 12:30:11.195638 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.195569    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hmqqr_must-gather-wh6nx_e5c64d46-806e-4a4f-813b-63d6d7f5643e/gather/0.log"
Jul 07 12:30:11.625926 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.625897    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:30:11.754545 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754514    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-56zxf/must-gather-plvzk"]
Jul 07 12:30:11.754858 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754843    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container"
Jul 07 12:30:11.754932 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754860    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container"
Jul 07 12:30:11.754932 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754877    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kube-rbac-proxy"
Jul 07 12:30:11.754932 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754886    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kube-rbac-proxy"
Jul 07 12:30:11.754932 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754899    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62"
Jul 07 12:30:11.754932 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754907    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62"
Jul 07 12:30:11.755191 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754980    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kserve-container"
Jul 07 12:30:11.755191 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.754994    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="91d36dfe-1452-4cdb-8e4e-59e877c9a71e" containerName="kube-rbac-proxy"
Jul 07 12:30:11.755191 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.755008    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" containerName="switch-graph-beb62"
Jul 07 12:30:11.757619 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.757600    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56zxf/must-gather-plvzk"
Jul 07 12:30:11.760007 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.759984    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-56zxf\"/\"openshift-service-ca.crt\""
Jul 07 12:30:11.760110 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.760021    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-56zxf\"/\"kube-root-ca.crt\""
Jul 07 12:30:11.761011 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.760993    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-56zxf\"/\"default-dockercfg-nl56f\""
Jul 07 12:30:11.767931 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.767912    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-56zxf/must-gather-plvzk"]
Jul 07 12:30:11.774129 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.774110    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:30:11.828862 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.828834    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwh9j\" (UniqueName: \"kubernetes.io/projected/0a23306b-0d5e-493e-855e-afb73702d5e0-kube-api-access-gwh9j\") pod \"must-gather-plvzk\" (UID: \"0a23306b-0d5e-493e-855e-afb73702d5e0\") " pod="openshift-must-gather-56zxf/must-gather-plvzk"
Jul 07 12:30:11.828956 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.828882    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a23306b-0d5e-493e-855e-afb73702d5e0-must-gather-output\") pod \"must-gather-plvzk\" (UID: \"0a23306b-0d5e-493e-855e-afb73702d5e0\") " pod="openshift-must-gather-56zxf/must-gather-plvzk"
Jul 07 12:30:11.839020 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.838996    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:30:11.930225 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.930168    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a23306b-0d5e-493e-855e-afb73702d5e0-must-gather-output\") pod \"must-gather-plvzk\" (UID: \"0a23306b-0d5e-493e-855e-afb73702d5e0\") " pod="openshift-must-gather-56zxf/must-gather-plvzk"
Jul 07 12:30:11.930301 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.930224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwh9j\" (UniqueName: \"kubernetes.io/projected/0a23306b-0d5e-493e-855e-afb73702d5e0-kube-api-access-gwh9j\") pod \"must-gather-plvzk\" (UID: \"0a23306b-0d5e-493e-855e-afb73702d5e0\") " pod="openshift-must-gather-56zxf/must-gather-plvzk"
Jul 07 12:30:11.930468 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.930452    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a23306b-0d5e-493e-855e-afb73702d5e0-must-gather-output\") pod \"must-gather-plvzk\" (UID: \"0a23306b-0d5e-493e-855e-afb73702d5e0\") " pod="openshift-must-gather-56zxf/must-gather-plvzk"
Jul 07 12:30:11.938202 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:11.938174    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwh9j\" (UniqueName: \"kubernetes.io/projected/0a23306b-0d5e-493e-855e-afb73702d5e0-kube-api-access-gwh9j\") pod \"must-gather-plvzk\" (UID: \"0a23306b-0d5e-493e-855e-afb73702d5e0\") " pod="openshift-must-gather-56zxf/must-gather-plvzk"
Jul 07 12:30:12.066609 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:12.066589    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56zxf/must-gather-plvzk"
Jul 07 12:30:12.186549 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:12.186526    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-56zxf/must-gather-plvzk"]
Jul 07 12:30:12.188997 ip-10-0-140-177 kubenswrapper[2572]: W0707 12:30:12.188972    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a23306b_0d5e_493e_855e_afb73702d5e0.slice/crio-8ff62cd6374ca7f803d5654a32f84a30b5dc8817a79548f9258493d34a5191f6 WatchSource:0}: Error finding container 8ff62cd6374ca7f803d5654a32f84a30b5dc8817a79548f9258493d34a5191f6: Status 404 returned error can't find the container with id 8ff62cd6374ca7f803d5654a32f84a30b5dc8817a79548f9258493d34a5191f6
Jul 07 12:30:12.437870 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:12.437812    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655adc5c-0a5b-43bf-a894-6743b04a0eba" path="/var/lib/kubelet/pods/655adc5c-0a5b-43bf-a894-6743b04a0eba/volumes"
Jul 07 12:30:13.020830 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:13.020803    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56zxf/must-gather-plvzk" event={"ID":"0a23306b-0d5e-493e-855e-afb73702d5e0","Type":"ContainerStarted","Data":"8ff62cd6374ca7f803d5654a32f84a30b5dc8817a79548f9258493d34a5191f6"}
Jul 07 12:30:14.026692 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:14.026638    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56zxf/must-gather-plvzk" event={"ID":"0a23306b-0d5e-493e-855e-afb73702d5e0","Type":"ContainerStarted","Data":"0d1dff4b17f358f20fac20d30aeb16d4c432134d495da3b1e182e60b751ff213"}
Jul 07 12:30:14.027115 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:14.026704    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56zxf/must-gather-plvzk" event={"ID":"0a23306b-0d5e-493e-855e-afb73702d5e0","Type":"ContainerStarted","Data":"769652999f6b836a77d5c3754332e7c5171141d851e2de270efe4c71ff85e962"}
Jul 07 12:30:14.043699 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:14.043642    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-56zxf/must-gather-plvzk" podStartSLOduration=2.2745885980000002 podStartE2EDuration="3.04362661s" totalImagesPullingTime="769.038012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:30:14.042331255 +0000 UTC m=+3000.237908305" watchObservedRunningTime="2026-07-07 12:30:14.04362661 +0000 UTC m=+3000.239203632"
Jul 07 12:30:14.404469 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:14.404390    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wrrg9_66a5788d-0db4-4335-ae38-7e916eca3b86/global-pull-secret-syncer/0.log"
Jul 07 12:30:14.461102 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:14.460993    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:30:14.476911 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:14.461243    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:30:14.501068 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:14.501040    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log"
Jul 07 12:30:14.659570 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:14.659491    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-177.ec2.internal_bf25b0953b6a4731ed485a31472da6c0/haproxy/0.log"
Jul 07 12:30:16.600399 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:16.600355    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-hmqqr/must-gather-wh6nx"]
Jul 07 12:30:16.600902 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:16.600590    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-hmqqr/must-gather-wh6nx" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerName="copy" containerID="cri-o://160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023" gracePeriod=2
Jul 07 12:30:16.602687 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:30:16.602637    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wh6nx\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hmqqr\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:30:16.603971 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:16.603943    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-hmqqr/must-gather-wh6nx"]
Jul 07 12:30:16.952806 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:16.952772    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hmqqr_must-gather-wh6nx_e5c64d46-806e-4a4f-813b-63d6d7f5643e/copy/0.log"
Jul 07 12:30:16.961800 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:16.961772    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:30:16.975453 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:30:16.969886    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wh6nx\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hmqqr\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:30:17.046375 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.046349    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hmqqr_must-gather-wh6nx_e5c64d46-806e-4a4f-813b-63d6d7f5643e/copy/0.log"
Jul 07 12:30:17.047039 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.047010    2572 generic.go:356] "Generic (PLEG): container finished" podID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerID="160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023" exitCode=143
Jul 07 12:30:17.047284 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.047271    2572 scope.go:117] "RemoveContainer" containerID="160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023"
Jul 07 12:30:17.047739 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.047723    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:30:17.050609 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:30:17.050577    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wh6nx\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hmqqr\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:30:17.062813 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.062796    2572 scope.go:117] "RemoveContainer" containerID="59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82"
Jul 07 12:30:17.078118 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.078095    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlnt8\" (UniqueName: \"kubernetes.io/projected/e5c64d46-806e-4a4f-813b-63d6d7f5643e-kube-api-access-vlnt8\") pod \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\" (UID: \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\") "
Jul 07 12:30:17.078637 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.078619    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5c64d46-806e-4a4f-813b-63d6d7f5643e-must-gather-output\") pod \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\" (UID: \"e5c64d46-806e-4a4f-813b-63d6d7f5643e\") "
Jul 07 12:30:17.081476 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.080140    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c64d46-806e-4a4f-813b-63d6d7f5643e-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e5c64d46-806e-4a4f-813b-63d6d7f5643e" (UID: "e5c64d46-806e-4a4f-813b-63d6d7f5643e"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:30:17.083875 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.083849    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c64d46-806e-4a4f-813b-63d6d7f5643e-kube-api-access-vlnt8" (OuterVolumeSpecName: "kube-api-access-vlnt8") pod "e5c64d46-806e-4a4f-813b-63d6d7f5643e" (UID: "e5c64d46-806e-4a4f-813b-63d6d7f5643e"). InnerVolumeSpecName "kube-api-access-vlnt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:30:17.085824 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.085186    2572 scope.go:117] "RemoveContainer" containerID="160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023"
Jul 07 12:30:17.085824 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:30:17.085587    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023\": container with ID starting with 160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023 not found: ID does not exist" containerID="160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023"
Jul 07 12:30:17.085824 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.085626    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023"} err="failed to get container status \"160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023\": rpc error: code = NotFound desc = could not find container \"160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023\": container with ID starting with 160613793f0ea3b90f76d4a134a1bb14d83843afbf9be5d96dad78438f994023 not found: ID does not exist"
Jul 07 12:30:17.085824 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.085652    2572 scope.go:117] "RemoveContainer" containerID="59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82"
Jul 07 12:30:17.086071 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:30:17.085944    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82\": container with ID starting with 59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82 not found: ID does not exist" containerID="59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82"
Jul 07 12:30:17.086071 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.085972    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82"} err="failed to get container status \"59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82\": rpc error: code = NotFound desc = could not find container \"59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82\": container with ID starting with 59839b6d89953ea3c2839363986d113f39d2d19faa304bbc4f3b899cb508fb82 not found: ID does not exist"
Jul 07 12:30:17.181223 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.181125    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlnt8\" (UniqueName: \"kubernetes.io/projected/e5c64d46-806e-4a4f-813b-63d6d7f5643e-kube-api-access-vlnt8\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:30:17.181223 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.181168    2572 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5c64d46-806e-4a4f-813b-63d6d7f5643e-must-gather-output\") on node \"ip-10-0-140-177.ec2.internal\" DevicePath \"\""
Jul 07 12:30:17.313954 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.313915    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:30:17.365371 ip-10-0-140-177 kubenswrapper[2572]: E0707 12:30:17.365332    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wh6nx\" is forbidden: User \"system:node:ip-10-0-140-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hmqqr\": no relationship found between node 'ip-10-0-140-177.ec2.internal' and this object" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" pod="openshift-must-gather-hmqqr/must-gather-wh6nx"
Jul 07 12:30:17.426572 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:17.426537    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-465p6_5bb35b90-7072-43a8-81de-f8db577cd975/konnectivity-agent/0.log" limit=1024
Jul 07 12:30:18.465151 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:18.465109    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" path="/var/lib/kubelet/pods/e5c64d46-806e-4a4f-813b-63d6d7f5643e/volumes"
Jul 07 12:30:19.204007 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.203963    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-lrrbt_4057a7ba-08cd-41d6-852e-25500d526c6f/kube-state-metrics/0.log"
Jul 07 12:30:19.227693 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.227666    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-lrrbt_4057a7ba-08cd-41d6-852e-25500d526c6f/kube-rbac-proxy-main/0.log"
Jul 07 12:30:19.252779 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.252736    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-lrrbt_4057a7ba-08cd-41d6-852e-25500d526c6f/kube-rbac-proxy-self/0.log"
Jul 07 12:30:19.317559 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.317523    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-k87rg_9a228e1c-da62-4d64-ae7c-70999bf0e97e/monitoring-plugin/0.log"
Jul 07 12:30:19.528888 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.528856    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n99nd_eec050d3-2cc3-4dca-82cd-e10bac52d313/node-exporter/0.log"
Jul 07 12:30:19.551476 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.551446    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n99nd_eec050d3-2cc3-4dca-82cd-e10bac52d313/kube-rbac-proxy/0.log"
Jul 07 12:30:19.580537 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.580503    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n99nd_eec050d3-2cc3-4dca-82cd-e10bac52d313/init-textfile/0.log"
Jul 07 12:30:19.700381 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.700343    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17fc6f49-0c45-4c5e-8613-c6e4c3da58fa/prometheus/0.log"
Jul 07 12:30:19.724838 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.724808    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17fc6f49-0c45-4c5e-8613-c6e4c3da58fa/config-reloader/0.log"
Jul 07 12:30:19.751122 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.751093    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17fc6f49-0c45-4c5e-8613-c6e4c3da58fa/thanos-sidecar/0.log"
Jul 07 12:30:19.775413 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.775382    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17fc6f49-0c45-4c5e-8613-c6e4c3da58fa/kube-rbac-proxy-web/0.log"
Jul 07 12:30:19.802335 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.802259    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17fc6f49-0c45-4c5e-8613-c6e4c3da58fa/kube-rbac-proxy/0.log"
Jul 07 12:30:19.829749 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.829718    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17fc6f49-0c45-4c5e-8613-c6e4c3da58fa/kube-rbac-proxy-thanos/0.log"
Jul 07 12:30:19.857038 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.857010    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17fc6f49-0c45-4c5e-8613-c6e4c3da58fa/init-config-reloader/0.log"
Jul 07 12:30:19.891464 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.891432    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-8lqww_2ea85072-891a-4c27-b309-af9dc0ffedaa/prometheus-operator/0.log"
Jul 07 12:30:19.912456 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:19.912427    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-8lqww_2ea85072-891a-4c27-b309-af9dc0ffedaa/kube-rbac-proxy/0.log"
Jul 07 12:30:20.092319 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:20.092233    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db8f4c79-rm4kf_4fc39e06-3f4d-40bd-89d2-32af355174ac/thanos-query/0.log"
Jul 07 12:30:20.121757 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:20.121726    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db8f4c79-rm4kf_4fc39e06-3f4d-40bd-89d2-32af355174ac/kube-rbac-proxy-web/0.log"
Jul 07 12:30:20.146349 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:20.146319    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db8f4c79-rm4kf_4fc39e06-3f4d-40bd-89d2-32af355174ac/kube-rbac-proxy/0.log"
Jul 07 12:30:20.184203 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:20.184163    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db8f4c79-rm4kf_4fc39e06-3f4d-40bd-89d2-32af355174ac/prom-label-proxy/0.log"
Jul 07 12:30:20.223283 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:20.223256    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db8f4c79-rm4kf_4fc39e06-3f4d-40bd-89d2-32af355174ac/kube-rbac-proxy-rules/0.log"
Jul 07 12:30:20.256128 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:20.256097    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db8f4c79-rm4kf_4fc39e06-3f4d-40bd-89d2-32af355174ac/kube-rbac-proxy-metrics/0.log"
Jul 07 12:30:22.381989 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.381954    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"]
Jul 07 12:30:22.382579 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.382350    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerName="gather"
Jul 07 12:30:22.382579 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.382370    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerName="gather"
Jul 07 12:30:22.382579 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.382401    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerName="copy"
Jul 07 12:30:22.382579 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.382411    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerName="copy"
Jul 07 12:30:22.382579 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.382477    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerName="copy"
Jul 07 12:30:22.382579 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.382488    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c64d46-806e-4a4f-813b-63d6d7f5643e" containerName="gather"
Jul 07 12:30:22.387958 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.387937    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.392019 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.391984    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"]
Jul 07 12:30:22.428303 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.428276    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-lib-modules\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.428449 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.428324    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-podres\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.428449 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.428370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-proc\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.428449 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.428400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsk2m\" (UniqueName: \"kubernetes.io/projected/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-kube-api-access-bsk2m\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.428552 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.428475    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-sys\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529058 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-sys\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529227 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529063    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-lib-modules\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529227 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-podres\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529227 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529144    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-proc\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529227 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529154    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-sys\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529227 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529211    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-proc\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529439 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsk2m\" (UniqueName: \"kubernetes.io/projected/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-kube-api-access-bsk2m\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529439 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529267    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-podres\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.529439 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.529289    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-lib-modules\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.537770 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.537747    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsk2m\" (UniqueName: \"kubernetes.io/projected/a4e91dfb-8b57-41aa-8110-e6f2f3c34655-kube-api-access-bsk2m\") pod \"perf-node-gather-daemonset-n9ghc\" (UID: \"a4e91dfb-8b57-41aa-8110-e6f2f3c34655\") " pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.700326 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.700254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:22.860159 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:22.859807    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"]
Jul 07 12:30:23.081727 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:23.081684    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc" event={"ID":"a4e91dfb-8b57-41aa-8110-e6f2f3c34655","Type":"ContainerStarted","Data":"e9e366b04310a00f4eecf40b2665de5b0c80e090beba57e08804c7ba68b36f36"}
Jul 07 12:30:23.081914 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:23.081743    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc" event={"ID":"a4e91dfb-8b57-41aa-8110-e6f2f3c34655","Type":"ContainerStarted","Data":"4cbeae4caed9dbea16759e7d23e55708f9d442ecb49b0ef9179335dfdbe30a19"}
Jul 07 12:30:23.099585 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:23.099526    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc" podStartSLOduration=1.099506203 podStartE2EDuration="1.099506203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 12:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:30:23.098851787 +0000 UTC m=+3009.294428809" watchObservedRunningTime="2026-07-07 12:30:23.099506203 +0000 UTC m=+3009.295083226"
Jul 07 12:30:23.686142 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:23.686110    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7r6ns_87c9f985-a645-4f3a-a284-1f515c5cb2a2/dns/0.log"
Jul 07 12:30:23.708005 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:23.707977    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7r6ns_87c9f985-a645-4f3a-a284-1f515c5cb2a2/kube-rbac-proxy/0.log"
Jul 07 12:30:23.926553 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:23.926522    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2rkx_5d9ceae5-ab86-4d83-b0ff-c5912fee77cd/dns-node-resolver/0.log"
Jul 07 12:30:24.085431 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:24.085388    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:24.498127 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:24.498085    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zh7g2_cf3e2930-9990-4025-a22b-b53af21851e7/node-ca/0.log"
Jul 07 12:30:25.630177 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:25.630147    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bxcm5_dd7c56e7-36b1-41e8-9624-2ca61840b3f3/serve-healthcheck-canary/0.log"
Jul 07 12:30:26.035266 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:26.035238    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2tjsn_b954c61a-ecae-42d2-9fd3-0c931f5cda90/kube-rbac-proxy/0.log"
Jul 07 12:30:26.056548 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:26.056524    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2tjsn_b954c61a-ecae-42d2-9fd3-0c931f5cda90/exporter/0.log"
Jul 07 12:30:26.077088 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:26.077067    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2tjsn_b954c61a-ecae-42d2-9fd3-0c931f5cda90/extractor/0.log"
Jul 07 12:30:28.411651 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:28.411626    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-zpxjl_7a4d2ee5-014d-479f-b760-51fe6624f419/manager/0.log"
Jul 07 12:30:28.459817 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:28.459759    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lhqjf_e920a33f-a9cd-4fd7-bb62-3507041a218c/s3-init/0.log"
Jul 07 12:30:28.474676 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:28.474645    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lhqjf_e920a33f-a9cd-4fd7-bb62-3507041a218c/download-hf-model/0.log"
Jul 07 12:30:30.098969 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:30.098943    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-56zxf/perf-node-gather-daemonset-n9ghc"
Jul 07 12:30:34.179373 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.179299    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nm6fk_71053fbc-7333-4971-9c42-0d84b4fb6a6b/kube-multus-additional-cni-plugins/0.log"
Jul 07 12:30:34.200851 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.200821    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nm6fk_71053fbc-7333-4971-9c42-0d84b4fb6a6b/egress-router-binary-copy/0.log"
Jul 07 12:30:34.222440 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.222418    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nm6fk_71053fbc-7333-4971-9c42-0d84b4fb6a6b/cni-plugins/0.log"
Jul 07 12:30:34.245305 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.245285    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nm6fk_71053fbc-7333-4971-9c42-0d84b4fb6a6b/bond-cni-plugin/0.log"
Jul 07 12:30:34.269381 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.269365    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nm6fk_71053fbc-7333-4971-9c42-0d84b4fb6a6b/routeoverride-cni/0.log"
Jul 07 12:30:34.289779 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.289758    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nm6fk_71053fbc-7333-4971-9c42-0d84b4fb6a6b/whereabouts-cni-bincopy/0.log"
Jul 07 12:30:34.313868 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.313851    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nm6fk_71053fbc-7333-4971-9c42-0d84b4fb6a6b/whereabouts-cni/0.log"
Jul 07 12:30:34.354471 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.354444    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fpndh_abea6595-66e7-4e5a-8414-e656fbe0b242/kube-multus/0.log"
Jul 07 12:30:34.493408 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.493323    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nwrkz_d622f883-4224-4801-b1d3-2b543c30dc35/network-metrics-daemon/0.log"
Jul 07 12:30:34.513383 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:34.513353    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nwrkz_d622f883-4224-4801-b1d3-2b543c30dc35/kube-rbac-proxy/0.log"
Jul 07 12:30:35.628432 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.628395    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-controller/0.log"
Jul 07 12:30:35.648325 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.648303    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/0.log"
Jul 07 12:30:35.648450 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.648324    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovn-acl-logging/1.log"
Jul 07 12:30:35.665892 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.665874    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/kube-rbac-proxy-node/0.log"
Jul 07 12:30:35.687488 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.687470    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 12:30:35.705608 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.705590    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/northd/0.log"
Jul 07 12:30:35.724119 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.724098    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/nbdb/0.log"
Jul 07 12:30:35.747821 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.747800    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/sbdb/0.log"
Jul 07 12:30:35.880790 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:35.880726    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpspv_cad78671-89ed-4a34-877e-7e0448976883/ovnkube-controller/0.log"
Jul 07 12:30:36.928450 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:36.928416    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zzbqh_0971da9a-0556-4091-a7de-253ec8dce3e4/network-check-target-container/0.log"
Jul 07 12:30:37.799792 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:37.799763    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bb9dr_c9602c0b-14d5-4878-9bb3-1b8914003013/iptables-alerter/0.log"
Jul 07 12:30:38.581580 ip-10-0-140-177 kubenswrapper[2572]: I0707 12:30:38.581550    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xdn6h_da24ad97-d836-4d7a-9db2-586d261e3349/tuned/0.log"